Lifetime problem in Rust/C wrapper: informing Rust of lifetime related to C opaque objects

help
1

I found myself with a problem that, from googling around, seems frequent: Rust complaining about a missing 'static when storing a dyn FnMut(...) -> ... into a Box. More specifically this happened when interfacing a C library that works with callbacks. The two solutions I came up with however seem to silently break Rust's lifetime checks.

I use a solution presented in this stackoverflow answer, which I adapted hereafter:

use std::os::raw::c_void;
use std::mem;
use std::ptr;

/* ***********************************************************
 * Code generated by bindgen
 * ***********************************************************/

#[derive(Debug, Copy, Clone)]
pub struct context {
    _unused: [u8; 0],
}
pub type context_t = *mut context;
pub type callback_fn_t = Option<unsafe extern "C" fn(ctx: context_t, data: *mut c_void) -> i32>;
pub type free_fn_t = Option<unsafe extern "C" fn(ctx: context_t, data: *mut c_void) -> ()>;

/*
extern "C" {
    pub fn c_register_callback(ctx: context_t, cb: callback_fn_t, data: *mut c_void, free: free_fn_t);
    pub fn c_call_callback(ctx: context_t);
    pub fn c_free_context(ctx: context_t);
}
*/

/* ***********************************************************
 * Mock implementation of the C API
 * ***********************************************************/
static mut CONTEXT: context_t = ptr::null_mut() as *mut _;
static mut DATA: *mut c_void =  ptr::null_mut() as *mut _;
static mut CB: callback_fn_t = None;
static mut FREE: free_fn_t = None;

extern "C" fn c_register_callback(ctx: context_t, cb: callback_fn_t, data: *mut c_void, free: free_fn_t) {
    unsafe {
        CONTEXT = ctx;
        CB = cb;
        FREE = free;
        DATA = data;
    }
}

extern "C" fn c_call_callback(ctx: context_t) {
    unsafe {
        if let Some(cb) = CB {
            cb(ctx, DATA);
        }
    }
}

extern "C" fn c_free_context(ctx: context_t) {
    unsafe {
        if let Some(free) = FREE {
            free(ctx, DATA);
        }
    }
}

/* ***********************************************************
 * First version, using Box<Box<dyn FnMut(context_t) -> i32>>
 * ***********************************************************/

extern "C" fn rust_generic_callback(ctx: context_t, data: *mut c_void) -> i32 {
    let cb: &mut Box<dyn FnMut(context_t) -> i32> = unsafe { mem::transmute(data) };
    cb(ctx)
}

extern "C" fn rust_generic_free(ctx: context_t, data: *mut c_void) {
    let _: Box<Box<dyn FnMut(context_t) -> i32>> = unsafe { Box::from_raw(data as *mut _) };
}

fn rust_register_callback<F>(ctx: context_t, cb: F)
where F: FnMut(context_t) -> i32 {
    let cb: Box<Box<dyn FnMut(context_t) -> i32>> = Box::new(Box::new(cb));
    unsafe { c_register_callback(
        ctx, Some(rust_generic_callback),
        Box::into_raw(cb) as *mut _,
        Some(rust_generic_free))
    };
}

Note that contrary to the stack overflow question, I did not add + 'static to the F type in the definition of rust_register_callback. Rust does not complain about it, even though obviously not all callbacks would work fine. Example:

fn install_callback_with_shorter_lifetime(ctx: context_t) {
    let mut x = 42;
    let y = &mut x;
    let cb = move |ctx: context_t| -> i32 {
        println!("Hello from callback, value is {}", *y);
        *y = 45;
        *y
    };
    rust_register_callback(ctx, cb);
    unsafe { c_call_callback(ctx); }
}

fn main() {
    let ctx: context_t = ptr::null_mut() as *mut _;
    install_callback_with_shorter_lifetime(ctx);
    unsafe { c_call_callback(ctx); }
    unsafe { c_free_context(ctx); }
}

This prints 42 when c_call_callback is called within install_callback_with_shorter_lifetime, but garbage the second time it's called.

My question here is: why Rust didn't warn me about this potential lifetime issue?

Next, I tried the following:

/* ***********************************************************
 * Second version, wrapping ctx into a Context object
 * and wrapping callback in a MyCallback object
 * ***********************************************************/
struct MyCallback<'a> {
    pub cb: Box<dyn FnMut(context_t) -> i32 + 'a>,
}

struct Context {
    pub ctx: context_t  
}

extern "C" fn rust_generic_callback2(ctx: context_t, data: *mut c_void) -> i32 {
    let my_callback: &mut MyCallback = unsafe { mem::transmute(data) };
    (my_callback.cb)(ctx)
}

extern "C" fn rust_generic_free2(ctx: context_t, data: *mut c_void) {
    let _: Box<MyCallback> = unsafe { Box::from_raw(data as *mut _) };
}

impl Context {
    
    pub fn new() -> Self {
        Context { ctx: ptr::null_mut() as *mut _ }
    }
    
    pub fn register_callback<'a: 'b, 'b, F>(&'b self, cb: F)
    where F: FnMut(context_t) -> i32 + 'a {
        let cb = MyCallback { cb: Box::new(cb) };
        let cb = Box::new(cb);
        unsafe { c_register_callback(
            self.ctx, Some(rust_generic_callback2),
            Box::into_raw(cb) as *mut _,
            Some(rust_generic_free2))
        };
    }
    
    pub fn call_callback(&self) {
        c_call_callback(self.ctx);
    }
}

impl Drop for Context {
    
    fn drop(&mut self) {
        c_free_context(self.ctx);
    }
}

My initial version did not have lifetime specifiers (in MyCallback and in register_callback), and Rust was complaining about missing a lifetime specifier, and suggesting + 'static. I had to read a lot of tutorials and posts on this forum to understand what this meant and realized "I don't want a 'static lifetime, I know that the callback is not going to be called after the context is destroyed, I want a lifetime that reflects that". Hence I added a lifetime 'b for the context 'a for the closure, which must outlive 'b. Adding the 'a and 'b lifetimes solved the compilation error, however it does not seem to solve the problem. I can still do something like this without any warning from Rust:

fn f(ctx: &Context) {
    let mut x = 42;
    let y = &mut x;
    let cb3 = move |ctx: context_t| -> i32 {
        println!("Hello from callback, value is {}", *y);
        *y = 45;
        *y
    };
    ctx.register_callback(cb3);
    ctx.call_callback();
}

fn main() {
    let ctx: Context = Context::new();
    f(&ctx);
    ctx.call_callback();
}

Once again, the first print statement works fine, while the second gives garbage data for the value of *y.
Once again, my question is: what prevented Rust from detecting that I am doing something invalid? And how do I make it such that I can't do such a thing?

Note: There are a few calls to "unsafe" when calling the C functions, but the problem is not down to those unsafe codes. Most of them can actually be removed when using the Rust-based mock version of the C API.

2

So remove them.

The one that cannot be removed without rustc yelling at you? That's the one that prevented the compiler detecting what you're doing wrong.

unsafe means "don't check this, I know what I'm doing." Rust isn't responsible for what happens when you lie to it.

4 Likes
3

did I silently break Rust’s checks?

No. While you are still learning the language, it's the safest thing to assume that you have not found a compiler bug.

Unsurprisingly, you haven't found a compiler bug in this case, either. You are just missing the crucial pieces of information that:

  1. fn() function pointers are always implicitly &'static(-ish); if they had regular reference syntax &'_ fn(…), then you could never get an &'a fn except for &'static fn. (Currently, there isn't even any syntax for describing a function pointer's lifetime.)
  2. When doing transmutations, either explicitly or by means of raw pointer casting (both of which you do), you can change lifetimes. Raw pointers have no lifetime annotations (that is why they are raw), and transmute is explicitly telling the compiler to treat one type as another.

Based on these (the lack of relevant lifetime annotations and you explicitly telling the compiler to trust it), it is simply unreasonable to ask for help from the compiler.

If you want your lifetimes to be checked, then don't use unsafe and raw pointers. It's as simple as that.

1 Like
4

The title was more of a clickbait "obviously I must have done something wrong myself" :slight_smile:

I think I now get the point that transmuting to raw pointer is where I loose Rust: it has no way to know that the context is storing the callback, because of this. If I try to write a full Rust code, where the Context object has to effectively store the callback, Rust starts complaining about lifetime. Here is an example:

use std::cell::RefCell;

struct Context<'a> {
    cb: RefCell<Option<Callback<'a>>>
}

struct Callback<'a> {
    cb: Box<dyn FnMut(&Context) -> i32 + 'a>
}

impl<'a> Context<'a> {
    
    pub fn install_callback<F>(&self, cb: F)
    where F: FnMut(&Context) -> i32 + 'a {
        let cb = Callback { cb: Box::new(cb) };
        self.cb.replace(Some(cb));
    }
}

fn f(ctx: &Context) {
    let mut x = 42;
    let y = &mut x;
    let cb = |_ctx: &Context| { *y };
    ctx.install_callback(cb);
}

fn main() {
    let ctx = Context {cb: RefCell::new(None) };
    f(&ctx);
}

So now my question is: how can I rewrite the code based on the C API so that it correctly conveys the information about the lifetime relationship (i.e. so that my Context and MyCallback structs end up safe for users)?

5

You are asking how to write a safe wrapper for your C API. This would require the person answering your question to come up with the correct function signatures. Function signatures in Rust clearly define the safety requirements of a function, i.e. how it is or isn’t allowed to be called, and those are enforced by the compiler. However, C function signatures do absolutely not do the same. As far as I can tell, you never explained this C API to us. As far as I’m concerned, all I’m seeing is 3 C functions without any documentation.

Without any documentation, I would assume that in the worst case, every call to any of those C functions will immediately trigger undefined behavior. If that would be true, there’s no point wrapping such a function in Rust, and no way to do it in a non-unsafe function.

If this conservative approximation happens to be wrong… well… without any documentation, or clear definition wich way these functions are or aren’t safe to call, and when and under which conditions they might possibly call which of the callbacks provided with which arguments, there’s no way to give you a safe wrapper in Rust.

Edit: Oh, on third read through the previous posts, I finally stumbled across the “mock implementation” of the C API. Is that exact implementation supposed to be behaving to exactly like the actual implementation, for the purpose of safety? If that was the case, the first thing to address in the design of a safe Rust wrapper would probably be thread-safety.

1 Like
6

Yes, this is pretty much a good model of what the C API is doing (except not with static variables; the callback is stored in the context_t, and the whole thing is protected by a mutex).

Right now I'm reading through PhantomData, which might solve my problem if I can add such a field to the Context object to hold a lifetime.

EDIT: I'm sure I'm really close to a solution with the following code: Rust Playground
But Rust is still not complaining when I want it to. Any suggestions?

7

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.