Lifetime in a module system


#1

Hi All,

I do have a quite interesting problem and I would like your input.

I am writing a module (RediSQL) for an external database (Redis).

Redis ensure me that some data are valid, they are allocated by Redis, pass me as a pointer and when I have done I leave the control to Redis that deallocate everything.

This does not play very well with the lifetime concept in Rust.

Up to now I simply avoided: I copied everything, did my work on the copy, deallocate the copy, and let Redis deallocate the original.

This worked quite well, but now I need to avoid all those memory allocations doing exactly the same I was doing before.

Where should I look? How can I convince the type checker that my references are valid?

Thanks,
Simone


#2

Can you elaborate on this part? You can typically bind a lifetime to a reference (created from a foreign ptr) and then the rest of the Rust code “anchors” against that lifetime. As long as the foreign ptr stays valid for that lifetime then things work well.

Perhaps you can show some code that you’re having trouble with.


#3

If you’re storing raw pointers from Redis, or Redis holds on to data Rust can’t track, you can use PhantomData to pretend it’s stored on Rust’s side and have the borrow checker track it.


#4

You might want to leave Redis’s automatic memory management (RedisModule_AutoMemory) disabled, and instead wrap Redis-allocated values in Rust wrappers that call the appropriate Free functions in their Drop implementations. Then these Rust values can “own” the values. A method can take a reference to the owner and give out a reference to the underlying value, knowing that it can’t be freed while the owner is borrowed.


#5

Hi All,

sorry for the late reply.

Let me recap the issue.

Redis provides data (string) that I need to read, Redis is completely responsible for them and I do not have to do anything to them except reading. So no allocation, no free, no manipulation.

Anyhow, in some way I need them to be available to my code, ideally using references and without copy them.

We are provide with an handy method: RedisModule_StringPtrLen that returns to me the beginning of the string and its length.

My first approach was something like this:

pub fn string_ptr_len(str: *mut rm::ffi::RedisModuleString)
                      -> String {
    unsafe {
            CStr::from_ptr(rm::ffi::RedisModule_StringPtrLen
                           .unwrap()(str, std::ptr::null_mut()))
                .to_string_lossy()
                .into_owned()
    }
}

And this worked great. However here I am copying the whole string.

Another attempt was something like this:


pub fn string_ptr_len(str: *mut rm::ffi::RedisModuleString)
                      -> &str {
    unsafe {
        let mut len = 0;
        let base = rm::ffi::RedisModule_StringPtrLen
            .unwrap()(str, &mut len) as *mut u8;
        let mut slice = slice::from_raw_parts(base, len);
        str::from_utf8_unchecked(slice);
    }
}

This would be ideal, I got a reference to the string, I can only read them and it is zero copy, wonderful.

However, it is required a lifetime for these references, and I don’t know what lifetime I should provide. (Here should I try to wrap everything into some type and use a PhatomType?)

Finally another approach that I tried is something like:

pub fn string_ptr_len(str: *mut rm::ffi::RedisModuleString)
                      -> String {
    unsafe {
        let mut len = 0;
        let base = rm::ffi::RedisModule_StringPtrLen
            .unwrap()(str, &mut len) as *mut u8;
        String::from_raw_parts(base, len, len)
    }
}

Here the problem is that I should forget the String to avoid double free (rust will try to free it somewhere) maybe I can work around this but it doesn’t seems very nice.
I would try something like a custom type that wrap the string and that inside the Drop simply forget about the string.

What I would really like is the second method that seems the most affine of what is happening down to the memory.

Do you guys have suggestions?


#6

I would think figuring out how redis memory management does it automaticity, is the way to determining where the lifetime belongs.


#7

It is simpler than that. I am assured that the strings are alive for all the time I need them. I just don’t know how to express this in rust.


#8

If that’s the case, I believe -> &'static str for your second option would be fine.


#9

As @parched said, it sounds like you can do:

pub unsafe fn string_ptr_len(str: *mut rm::ffi::RedisModuleString)
                      -> &'static str {
    
        let mut len = 0;
        let base = rm::ffi::RedisModule_StringPtrLen
            .unwrap()(str, &mut len) as *mut u8;
        let mut slice = slice::from_raw_parts(base, len);
        str::from_utf8_unchecked(slice)
    
}

You can also make the fn generic over a lifetime parameter and bind the returned slice to that lifetime. This basically means the caller selects the scope of this reference. Probably not useful here but just wanted to mention that.

Another option is to wrap the raw ptr in a struct that implements Deref<str> - this “anchors” the string in this struct, and doesn’t allow the slice to outlive it. If the string is truly 'static from the module perspective, then this is probably unnecessary.

Side note: your function should be unsafe itself.


#10

Thank you guys.

It actually worked and solved my problem while improving the performance quite a bit.

Here the results: https://github.com/RedBeardLab/rediSQL/issues/29#issuecomment-383585843

Basically double the performance :slight_smile:


#11

What approach did you end up going with?


#12

The second one, the one that you guys suggested: https://github.com/RedBeardLab/rediSQL/blob/type_refactor/redisql_lib/src/redis.rs#L410 (Still unclean code)

I had a related but different issue with lifetimes before.


#13

Ok, cool. As mentioned, don’t forget to mark the fn unsafe as a whole :slight_smile:. May also want to check the raw ptr for null before doing the slice dance.


#14

Yeah, about this, why?

I know that the invariant are respected, so why should I mark the function as unsafe?

Sorry for replying so late…


#15

If the function requires the caller to uphold an invariant that’s not enforceable by Rust, then the function itself should be unsafe. Here the caller passes you a raw ptr, so they’re responsible for making sure it points to a valid RedisModuleString. I realize you’re your own caller here, but I’d still follow the convention.