Can someone explain why this is working?

I was looking for wrapping struct with lifetime for FFI and found the old topic How to deal with lifetime when need to expose through FFI - #2 by Yandros

This actually worked for my use case as well! Amazing. However, I was reading the PhantomData in core::marker - Rust but I wasn't able to find the explaination for this use case (or I don't understand correctly). I am quite amazed we can cast FfiContext to Context<'_,'_> in the from_ptr function using &mut *(ptr.cast()).

How come this is possible? This looks like some dark magic to me. Does compiler actually map the memory layout FfiContext as if the Context<_,_> existed? Is it okay to relay on this? I am wondering if there is any good docs/blogs/explainations about this.

Thanks in advance!

Sorry, I'm not getting what exactly you are asking for or what you expect shouldn't be working? Can you add an actual, minimal code example that demonstrates your problem and your expectations, and the actual behavior you observe instead?

2 Likes

It's worth noting that this solution is fairly dangerous. You are essentially throwing away the lifetimes when you send it through FFI, and then inventing new ones when you cast it back to the rust struct. If you aren't careful it will be very easy to end up with invalid lifetimes that reference data that no longer exists. The compiler can't check that for you.

I think you may be thinking that something more sophisticated is happening there than what's actually happening. Nothing is really happening at all in terms of runtime behavior, it's just changing the type the compiler thinks is behind the pointer.

The PhantomData there is mostly unrelated to what you're trying to do. It helps ensure the compiler correctly adds (or does not add) auto traits like Sync and Send to the ffi struct. You could remove the PhantomData and it would still work fine, though it might have misleading auto trait impls which could cause problems.

1 Like
  • .cast() can convert any *mut T to any *mut U where U is Sized
    • So ptr.cast() turns the *mut FfiContext into *mut Context<'_, '_>
  • &mut * (...) is then just reborrowing to create a &mut Context<'_, '_>
    • The lifetimes created are arbitrary and it's up to you to make sure you don't use them incorrectly
    • And all of Rust's other guarantees like having no aliasing of what is reachable from the &mut
  • There's no mapping of memory or anything, these are all basically just type conversions of a pointer
  • It's all unsafe
4 Likes

Yeah, that's exactly the part being asked about that I don't get; it might also be what OP doesn't get. To sum up:

  • any raw pointer type can be converted to any other raw pointer type (except fat pointers to dynamically-sized types, but let's not get into that)
  • types aren't a thing at runtime; pointers are just memory addresses, so if you pass a pointer P of type *const T, the exact numeric value received by the function is the same as if you had passes a *mut T or a *const U or a *const () or whatever.
  • the C function that reads from the pointer has already been compiled and it doesn't care (or know) about Rust types. It will read from behind the pointer according to whatever type it assumed the pointer points to when that function was compiled. The way in which the function dereferences the pointer is baked into the executable code of the function; it doesn't depend on how you call it from any other langue. There is no need for "memory mapping" or whatever.
  • The type conversions in this piece of Rust code are basically only there to shut up the compiler.
4 Likes

Thank you all for the great answers. After reading through the thread, I think I got what it is actually happening now.

  1. PhantomData used in the example is just a type to make compiler happy so we can embed the type with lifetime in the FfiContext struct without having FfiContect to have a lifetime parameter.

  2. Since the casting is unsafe, it is basically similar in C, we can cast any pointer to anything (besides the case mentioned by @H2CO3 ), just like in c we can do *char -> void * -> RandomStruct * . This is dangerous and off course it is unsafe in Rust.

Eventually, the code was cast Box<Context> into a raw pointer FfiContext, and then cast the raw pointer FfiContext back to Context using unsafe code. I was somehow confused by the use of PhantomData. But I think I got it now.

Thank you all for the great explaination!

Sorry for the late answer (given that this thread stems from a snippet I authored) :sweat_smile:

I don't have much to add, just to confirm / express my agreement with what has already been pointed out by others (thanks @H2CO3, @quinedot and @semicoleon).

  • Indeed, the PhantomData in that example was there to try and get some of the autotraits (Send, Synd) right, or at least, less wrong.

I'd also want to draw attention to:

which ought not to be underestimated. The snippet:

is indeed yielding something with unbounded lifetimes (in this instance, all three '_0, '_1 and '_2 are unbounded).

Nowadays, I find it more cautious to try and cage so-produced data structures within the scope of the ffi function. A dirty / conceptually-absurd but convenient way to do this is to tie the returned lifetimes to the lifetime of some borrow of the pointer itself:

unsafe
fn from_ptr_bounded<'ptr> (
    &mut ptr: &'ptr mut (*mut FfiContext),
) -> &'ptr mut Context<'ptr, 'ptr>
{
    /* same body */
}
Example
#[no_mangle] pub unsafe extern "C"
fn update (mut p: *mut FfiContext)
  -> u32
{
    //        cannot outlive `p`, and thus, cannot escape this function!
    //        vv             vv  vv
    let ctx: &'_ mut Context<'_, '_> = from_ptr_bounded(&mut p);
    ctx.update(delta);
    0
}

now we have to call it as let ctx = from_ptr_bounded(&ptr);, and the returned ctx will be lifetime-bound to the lifetime of that ptr local variable, which itself cannot outlive the FFI function that works with it. So we successfully prevent said ctx from escaping the FFI function boundaries.

Now, in this case we have nested lifetimes, so this trick may be too restrictive (the nested lifetimes are non-covariant, etc.).

Thus, a middle-ground between this super-careful but maybe overly-restrictive approach, and the original completely unbounded one, would be to keep the inner lifetimes unbounded, but not the outer one:

unsafe
fn from_ptr_half_bounded<'ptr, '_1, '_2> (
    &mut ptr: &'ptr mut (*mut FfiContext),
) -> &'ptr mut Context<'_1, '_2>
//    ^^^^             ^^^  ^^^
//    bounded          unbounded
{
    /* same body */
}
Example
#[no_mangle] pub unsafe extern "C"
fn update (mut p: *mut FfiContext)
  -> u32
{
    //        cannot outlive `p`, and thus, cannot escape this function!
    //        vv
    let ctx: &'_ mut Context<'_, '_> = from_ptr_half_bounded(&mut p);
    //                       ^^  ^^
    //                       but the `Context` referee is now unbounded again, so if cloned or something like that it could escape.
    ctx.update(delta);
    0
}

Finally, if that "lifetime of the ptr local itself" hack looks too weird, another option would be to feature CPS / a scoped/callback-based API:

unsafe
fn with_ctx_from_ptr<R> (
    ptr: *mut FfiContext,
    // another bonus: we can keep all the lifetimes elided and It Just Works™
    scope: impl FnOnce(&'_ mut Context<'_, '_>) -> R,
) -> R
{
    let yield_ = scope; // calling the callback is like _yielding_ the value.
    if … { … }
 // return &mut *ptr.cast())
    yield_(&mut *ptr.cast())
}

Callers would then use it like this:

#[no_mangle] pub unsafe extern "C"
fn update (p: *mut FfiContext)
  -> u32
{
    with_ctx_from_ptr(p, |ctx: &'_ mut Context<'_, '_>| {
        //                      ^^             ^^  ^^
        //                 completely unnameable / unable to escape 💪
        ctx.update(delta);
    });
    0
}

which would be the less error prone API of them all, imho, and would not require using that hack with the lifetime of the ptr local itself.

6 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.