Passing Rust closures to C

help
1

Edit: Changed title from "Multiple closures with mutable references to the same variable?" to "Passing Rust closures to C".

I'm working on creating a sandbox for interpreted languages (including Lua) in Rust. In that context, I need to convert Rust closures to pointers to call them from within the scripting language. I followed the nice tutorial of @Michael-F-Bryan, Rust Closures in FFI. This works well!

But I'm a bit concerned regarding mutability. Consider the following (non-FFI) code:

fn consume_closure<F>(storage: &mut Vec<F>, mut func: F)
where
    F: FnMut(),
{
    func();
    storage.push(func);
}

fn main() {
    let mut storage1 = vec![];
    let mut storage2 = vec![];
    let mut number = 7;
    consume_closure(&mut storage1, || {
        let number = &mut number;
        println!("Closure 1: {}", number);
        *number += 1;
    });
    // The following works,
    // i.e. there must be a mutable reference to `number`:
    storage1[0]();
    storage1[0]();
    // But why can we create another?
    consume_closure(&mut storage2, || {
        let number = &mut number;
        println!("Closure 2: {}", number);
        *number += 1;
    });
    // There is an error if we try again:
    //storage1[0]();
}

(Playground)

It is possible to create two closures, of which each holds a mutable reference to number. Why does that work? Doesn't the Rust compiler ensure there is only one mutable reference at a time?

Maybe the compiler is smart and notices that the closure in storage1 isn't used, and throws an error if I attempt to use it. But what if the closure gets converted to a raw pointer and will be handled in C from then on?

fn consume_closure<F>(mut func: F)
where
    F: FnMut(),
{
    func();
    let raw = Box::into_raw(Box::new(func));
    println!("This goes to C land: {:?}", raw);
}

fn main() {
    let mut number = 7;
    consume_closure(|| {
        let number = &mut number;
        println!("Closure 1: {}", number);
        *number += 1;
    });
    consume_closure(|| {
        let number = &mut number;
        println!("Closure 2: {}", number);
        *number += 1;
    });
}

(Playground)

I don't get an error here, but I could use each of those raw pointers to later invoke the closure. Why? What am I missing (or misunderstanding)?

1 Like
2

Yes. This particular behavior was added to Rust after the initial design for convenience, and is known as "non-lexical lifetimes" (NLL). Specifically, the lifetime of the borrow of number extends only to the last point it is used, which is the second storage1[0]();

In that case, you've lied to the compiler by not adding the bound F: 'static to consume_closure(). Without a lifetime bound, you can't assume a type variable (or plain reference) is valid after the function returns, and in this case the raw pointer isn't valid in that way because of the mutable reference. (Even an immutable reference would be a problem; it isn't required to be unique but it is required to point to memory that hasn't been deallocated or mutated.)

11 Likes
3

Oh, I didn't know, I thought lifetimes always end depending on where a block ends, etc. as explained here in the reference. But apparently that section of the reference only applies to destructors then, and not to borrows? Where in the reference can I find information on NLLs? I assume it's not writen yet? Are NLLs stabilized at all? The tracking issue seems to be open yet.

:scream:

So my closure could access variables that are already dropped. Certainly need to fix that! Thanks for the notice!!

I'm still unsure what's best:

  • Consume a closure, or
  • Take a mutable reference to the closure.

I assume the latter doesn't give me any advantages (but only disadvantages), as a mutable reference would also need to be 'static.

However, I know that the closure won't be invoked after my virtual machine (that is the only thing which can call the closure) is dropped. Thus I wonder if I can somehow use a shorter lifetime. I'm thinking on the same mechanism that rayon and crossbeam use for scoped threads. I really would like to avoid having to use Arcs.

Anyway, thanks for the help so far!

1 Like
4

Yes, exactly. The end of a lifetime is not and will likely never be an event that can be observed at run time, so it's okay for the compiler to relocate it (as long as future versions of the compiler continue to accept programs previous versions did). On the other hand, it's critical that destructors run at highly predictable times — for example, so that you can use a raw pointer into a local variable and know it'll stay valid until the end of the block.

Correct. In particular, the standard library provides impl<F: FnMut(...)> FnMut(...) for &mut F, so if your code accepts FnMut, that means it also accepts mutable references to the same kind of function. Thus, fn foo<F: FnMut(...)>(f: F) is more general than fn foo<F: FnMut(...)>(f: &mut F).

The simplest implementation is:

  • You have a type struct VirtualMachine<'a> {...}, that has a lifetime parameter. (This implicitly requires that the lifetime outlives any particular instance of the type.)
  • When you accept a function type, you bound it with 'a, like
    impl<'a> VirtualMachine<'a> {
    fn something_wants_a_closure<F: FnMut() + 'a>(&self, f: F) {...}
}
  • All of the pointers the machine might use are dropped, or at least not ever used again, when the VirtualMachine struct is dropped.

This takes advantage of a pattern the borrow checker understands; even if you're using pointers internally, it looks just like "a struct containing references" from the outside.

The disadvantage of this strategy is that any closure passed in that borrows data must borrow data that exists for at least as long as the VirtualMachine does. This is entirely adequate if your machine is “one-shot” in some way, like executing one user input in a REPL, but not if you want the VM to be able to borrow shorter-lived data given to it by a normal Rust function calling it; you then need to do the same thing, creating another struct with lifetime that guarantees the pointers will be dropped when it is.

An alternative is that data the VM wants to refer to can be wrapped in Rc/Arc instead of & — then, the data can live as long as the VM needs it to, instead of obligating the VM to fit into the shape the borrow checker understands. This can be mixed with the VirtualMachine<'a> strategy.

3 Likes
5

Okay, that makes sense to me. Thanks for explaining the reasoning.

I think I noticed some time before that I may pass references to closures instead of passing the closure itself. I think that's exactly due to the above implementation.

So I will make my functions expect F, and not &mut F, and they will support both. Very nice!

I (wrongly) thought I can avoid that by simply using the lifetime of the reference passed to my function.

Let me share some of my wrong attempts (for those interested in it).

Wrong code #1:

struct Machine {}

impl Machine {
    fn push_closure<'a, F>(&'a mut self, closure: F)
    where
        F: FnMut() + 'a,
    {
        println!(
            "This goes to C land: {:?}",
            Box::into_raw(Box::new(closure))
        );
    }
    fn run_closures(&mut self) {
        println!("Using closures from C land.");
    }
}

fn main() {
    let mut x: i32 = 7;
    let mut machine = Machine {};
    machine.push_closure(|| x += 1);
    drop(x);
    machine.run_closures();
}

(Playground)

Output:

This goes to C land: 0x55f6d730c9d0
Using closures from C land.

The closure can be used after x is dropped, which is bad.

I was at first surprised that even if I make the mutable reference to machine live longer, I still get undesired behavior:

Wrong code #2:

struct Machine {}

impl Machine {
    fn push_closure<'a, F>(&'a mut self, closure: F)
    where
        F: FnMut() + 'a,
    {
        println!(
            "This goes to C land: {:?}",
            Box::into_raw(Box::new(closure))
        );
    }
    fn run_closures(&mut self) {
        println!("Using closures from C land.");
    }
}

fn main() {
    let mut x: i32 = 7;
    let mut machine = Machine {};
    let machine_ref = &mut machine;
    machine_ref.push_closure(|| x += 1);
    drop(x);
    machine_ref.run_closures();
    drop(machine_ref);
}

(Playground)

Output:

This goes to C land: 0x5629cc08f9d0
Using closures from C land.

I assume that is due to subtyping, i.e. the compiler will just use a shorter lifetime for 'a but accepts the longer-living reference to the machine (while the closure lives shorter).

But when I do it as you suggested, @kpreid, it works correctly:

Correct code with lifetime parameter:

use std::marker::PhantomData;

struct Machine<'a> {
    _phantom: PhantomData<&'a ()>,
}

impl<'a> Machine<'a> {
    fn push_closure<F>(&mut self, closure: F)
    where
        F: FnMut() + 'a,
    {
        println!(
            "This goes to C land: {:?}",
            Box::into_raw(Box::new(closure))
        );
    }
    fn run_closures(&mut self) {
        println!("Using closures from C land.");
    }
}

fn main() {
    let mut x: i32 = 7;
    let mut machine = Machine {
        _phantom: PhantomData,
    };
    machine.push_closure(|| x += 1);
    drop(x); // commenting out this line will fix the error
    machine.run_closures();
}

(Playground)

Errors:

   Compiling playground v0.0.1 (/playground)
error[E0503]: cannot use `x` because it was mutably borrowed
  --> src/main.rs:28:10
   |
27 |     machine.push_closure(|| x += 1);
   |                          -- - borrow occurs due to use of `x` in closure
   |                          |
   |                          borrow of `x` occurs here
28 |     drop(x); // commenting out this line will fix the error
   |          ^ use of borrowed `x`
29 |     machine.run_closures();
   |     ---------------------- borrow later used here

For more information about this error, try `rustc --explain E0503`.
error: could not compile `playground` due to previous error

It makes sense because the Rust compiler won't know about when I use the closures from within the scripting language. It could happen any time the machine still exists.

I suppose it would be something like this:

Using a guard to invalidate the closure:

(Edit: This is not sound, as pointed out by @kpreid below.)

struct ClosureGuard<'a> {
    _machine: &'a Machine,
    ptr: *mut (),
}

impl<'a> Drop for ClosureGuard<'a> {
    fn drop(&mut self) {
        println!(
            "Ensure that {:?} will not be called anymore.",
            self.ptr
        );
    }
}

struct Machine {}

impl Machine {
    fn push_closure<'a, 'b, F>(&'a self, closure: F) -> ClosureGuard<'b>
    where
        'a: 'b,
        F: FnMut() + 'b,
    {
        let ptr = Box::into_raw(Box::new(closure));
        let guard = ClosureGuard {
            _machine: self,
            ptr: ptr as *mut (),
        };
        println!("This goes to C land: {:?}", ptr);
        guard
    }
    fn run_closures(&self) -> Result<(), ()> {
        println!("Using closures from C land.");
        println!("(Throwing runtime error if invalidated closures are used.)");
        Ok(())
    }
}

fn main() {
    let mut x: i32 = 7;
    let machine = Machine {};
    let closure_guard = machine.push_closure(|| x += 1);
    // We can't drop `x` without causing an error:
    // drop(x);
    machine.run_closures().unwrap();
    drop(closure_guard);
}

(Playground)

Output:

This goes to C land: 0x563c0b3129d0
Using closures from C land.
(Throwing runtime error if invalidated closures are used.)
Ensure that 0x563c0b3129d0 will not be called anymore.

Summary:

That effectively gives me three choices:

  1. require Rust closures to be 'static when passing them to the virtual machine,
  2. add a lifetime parameter 'a to the virtual machine and require Rust closures to be 'a when passing them to the virtual machine,
  3. when passing a closure to the virtual machine, return a guard that must be kept alive for as long as the closure shall be used (dropping the guard will result in an invalidation of the closure inside the virtual machine, causing runtime/scripting errors if the closure is used afterwards). (Edit: This is not sound, as pointed out by @kpreid below.)

(or a combination thereof)

Not sure yet which way I would go, but I tend to use a combination of choice #2 and #3, i.e. I will extend the Machine with a lifetime parameter, and I will provide two functions to push closures to the virtual machine:

  • using the lifetime of the machine: impl<'a> Machine<'a> { fn push_lt<F: FnMut() + 'a>(&self, closure: F) { /* … */ } }
  • using the lifetime of a returned guard: impl Machine<'a> { fn push_grd<'b, 'c, F>(&'b self, closure: F) -> ClosureGuard<'c> where 'b: 'c, F: FnMut() + 'c { /* … */ } }

But I'm not sure yet what's the best way to go in my case.

P.S.: Perhaps it's better to use a struct with a name such as ClosureContext which could then provide a method push_closure. Then this context could serve as a guard for multiple closures, which might make the API a bit nicer. (I.e. dropping the ClosureContext will invalidate all closures created in that context.)

6

The borrow check has used NLL since Rust 2018. The best place to find detailed information on how it works is the RFC.

2 Likes
7

That doesn't work: any guard / drop glue can be disabled by mem::forgetting the handle (as a comparison point, most of the problems here with FFI are very similar to those of spawning threads and "joining" to unregister them; and as you can see, most thread spawning APIs require 'static).

The way to ensure an eventual / unavoidable cleanup (within some region 'lt) is to use

The scoped API (callback) pattern

  • Before anything, some helper(s) (e.g., to debug stuff):

    pub struct Deathrattle(pub &'static str);
impl Drop for Deathrattle {
    fn drop (self: &'_ mut Self)
    {
        println!("Dropping `{}`", self.0);
    }
}

Let's start with a callback whose body kind of defines a scope:

// Naïve approach
fn run<R> (f: impl FnOnce() -> R)
  -> R
{
    let ret = f();
    println!("(Right before) end of scope");
    ret
}

fn main ()
{
    let _: i32 = {
        let value = Deathrattle("value");
        run(|| {
            let _captured = &value;
            println!("Body");
            42
        })
    };  
}

which prints:

Body
(Right before) end of scope
Dropping `value`

Now, this code has an important issue as well: what if f() panics? Then we are technically "returning an unwind", with a short-circuiting behavior which leads to the "(Right before) end of scope" "cleanup" to be skipped!:

  • Playground which prints "Body" and then Dropping …, with no end of scope whatsover.

This is because of have written code after something, rather than relying on drop glue. Indeed, drop glue is the usual mechanism to avoid the footgun not to run cleanup glue on short-circuiting returns, such as panic!s or ?:

// Naïve approach
fn run<R> (f: impl FnOnce() -> R)
  -> R
{
+   let _guard = ::scopeguard::guard((), /* on drop: */ |()| {
+       println!("(Right before) end of scope")
+   });
    let ret = f();
-   println!("(Right before) end of scope");
    ret
}
  • (::scopeguard is the go-to crate for ad-hoc drop glue instances).

With this, we do get:

thread 'main' panicked at 'Body', src/main.rs:27:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
(Right before) end of scope
Dropping `value`
  • The interesting bit is that I've started this post warning against relying too much on drop glue, and yet here I am using drop glue! The difference lies in who owns the drop-glue-imbued handle: since an owner, but very definition, is the one who will may drop the value, we can trust the drop glue provided we, the API / the callee are the ones owning the drop-glue-imbued handle (hence the callback style), which is not the case with more straight-forward APIs which return guards rather than taking callbacks.

I've personally written a crate to make this pattern more readable, since this is, after-all, related to unwind safety (it's as basic / simple as just a builder-pattern wrapper around a scopeguard, but the increased readability is quite nice imho):

fn run<R> (f: impl FnOnce() -> R)
  -> R
{
    ::unwind_safe::with_state(())
        .try_eval(|_| f())
        .finally(|_| {
            println!("(Right before) end of scope");
        })
}

So, now that we have something resembling a scope, let's try to get a lifetime parameter with which to work: a 'scope so that anything that is : 'scope may not dangle before the unregistration / cleanup. Given that property, let's call it 'beyond_cleanup instead:

- fn run<R> (f: impl FnOnce() -> R)
+ fn run<'beyond_cleanup, R> (f: impl FnOnce() -> R)
    -> R

And that's it! No need to bound anything here (not even f: impl … + 'beyond_cleanup!). Indeed:

  • <'beyond_cleanup> is a(n external) generic function parameter. From the looks of it, it is unbounded. This means that it could be used to represent any region of code …
    provided that region span beyond the end of run! Indeed, that's a kind of implicit / unknown aspect of lifetime parameters "in scope": they must represent a lifetime that spans beyond the end of a function's body the moment they are in scope of that function (≠ for<'any> lifetimes, which appear in HRTB and are thus no longer in scope within a function's body).

    So there is an effective lower bound on 'beyond_cleanup, so that if, later on, we write T : 'beyond_cleanup, we'd effectively be lower-bounding T by this lifetime parameter which is free but itself lower-bounded to outlive / span beyond our end of scope cleanup.

  • f is consumed within the call to run, so by the very construction / design, shan't be alive beyond the end of the scope. So we could very well add a + 'beyond_cleanup bound on f (and many real-life scoped APIs do, for the sake of "documentation" / good measure, even if it's technically not needed).

Now, there isn't much to use 'beyond_cleanup with, however, with the current design: if we wanted to perform that .spawn() API taking some Machine<'beyond_cleanup> handle, as you had, we'll need to integrate that Machine in there. For the sake of generality, I'll call it ScopeHandle:

fn run<'beyond_cleanup, R> (
    f: impl FnOnce(&ScopeHandle<'beyond_cleanup>) -> R,
) -> R
{
    let scope: ScopeHandle<'beyond_cleanup> = ScopeHandle {
        _beyond_cleanup_lifetime: PhantomInvariant,
    };
    ::unwind_safe::with_state(scope)
        .try_eval(|scope: &'_ ScopeHandle<'beyond_cleanup>| {
            impl<'beyond_cleanup> ScopeHandle<'beyond_cleanup> {
                // `f` cannot dangle / contain references that dangle
                // before `'beyond_cleanup` ends
                fn spawn(&self, f: impl Fn() + 'beyond_cleanup)
                {
                    Box::leak(Box::new(f)); // etc.
                }
            }

            f(scope)
        })
        .finally(|scope| { // cleanup!
            drop(scope);
            println!("Cleanup!");
        }) // <- and `'beyond_cleanup` must necessarily span beyond this.
}

// where
struct ScopeHandle<'beyond_cleanup> {
    _beyond_cleanup_lifetime: PhantomInvariant<'beyond_drop>,
}

There is an importance nuance, here: that 'beyond_cleanup lifetime is used as a lower-bound for the area of owned-usability of f (the point of all this design). It's important that such lower-bound not be allowed to shrink[1]. The official phrasing for this property is that we don't want type ScopeHandle<'beyond_cleanup> to be covariant in 'beyond_cleanup. It should, at best, be contravariant (lower bound is allowed to grow), and, in practice, invariant (neither cov. nor contra.) is just saner: let's not allow it to do something we don't need it to be allowed to do (even if it would be harmless in this instance). Hence the PhantomInvariant

This takes care of having a 'beyond_cleanup lifetime lower bound for stuff such as closures or other items that may be used concurrently all the way up that region.

But now let's imagine we also want to use that ScopeHandle to generate new instances which are, themselves, lifetime infected with a lifetime parameter that isn't allowed to span beyond the start of the cleanup process, that is, a lifetime that, at most, spans 'until_start_of_cleanup. Well, when we think about this, the very borrow over the ScopeHandle itself is a very nice candidate / representation of such a lifetime! I had just kept it elided to alleviate the signature, but the full signature would be:

fn run<'beyond_cleanup, R, F> (
    f: F,
) -> R
where
    for<'until_cleanup>       // implicit `'beyond_cleanup : 'until_cleanup` bound
    F : FnOnce(&'until_cleanup ScopeHandle<'beyond_cleanup>) -> R,

As you can see, "a wild HRTB appeared", which shouldn't be surprising, given how I had mentioned that (classic) lifetime parameters necessarily spanned beyond the end of a function's body, and how here we wanted the very opposite with that 'until_cleanup lifetime.

The fact it is higher-order means that for a caller's closure to be eligible to become / to be usable as f, it needs to be "agnostic" in the lifetime of the borrow of the ScopeHandle; it needs to ignore / disregard it. Or to be precise, any attempt not to do so is doomed to fail, since the higher-orderness of the signature there "threatens" the actual 'until_cleanup to be arbitrarily / infinitely small / short (although beyond f's body itself, of course): such a lifetime won't be able to escape its scope.

And, as I mentioned initially, this is exactly what ::crossbeam's scoped-thread spawning (auto-joined) API is like:

fn scope<'env, F, R> (
    f: F,
) -> thread::Result<R>
where
    F : FnOnce(&Scope<'env>) -> R,
  • 'env is 'beyond_join, and thus represents the lifetime bound for the .spawn()-ed closure's 'environment.

Note that once we have an "owned thing" we construct in our scope API, we can try to embed all the cleanup within its drop glue, simplifying the implementation:

Final signature

fn scope_api<'beyond_scope, R, F> (
    f: F,
) -> R
where
    for<'scope> 
        F : FnOnce(&'scope Scope<'beyond_scope>) -> R
    ,
{
    let scope_handle: Scope<'beyond_scope> = …;

    impl Drop for Scope<'_> { fn drop(&mut self) {
        /* cleanup */
    }}

    f(&scope_handle)
}

  1. to illustrate: if somebody owes you some amount X of money (say 10), and you manage to guarantee that you'll be reimbursed an amount at least as big as X, you don't want that, amidst all that paperwork, X happen to shrink (say, become 5). ↩︎

4 Likes
8

This is not sound because callers can leak the guard, preventing its drop code from running. To enforce cleanup at a stack frame you have to instead write a function that takes a closure and immediately calls it — then the function is guaranteed to see a return or panic.

I would summarize it like so: In &'a mut SomeType<'b>, the implicit lifetime requirements are that 'b must be longer than (or equal to) the existence of this instance of SomeType, and 'a must be shorter (or equal). Both of these correspond to the direction of the references they're protecting: a reference must be outlived by its referent.

1 Like
9

@Yandros: Wanted to say thanks already for your detailed post. I'm still digesting/processing it.

That makes sense. When you don't get ownership of the handle, you can't mem::forget it.

… okay, that's unexpected, but very interesting.

1 Like
10

I'm still having a hard time following through your post. I haven't worked with scopeguard yet, but I understand what it's supposed to do. But where I can't follow is what a PhantomInvariant is, for example.

I also don't understand why there is an impl Drop within a function body in the "Final signature".

Anyway, I tried to put those pieces of your post which I did understand together, and this is what I came up with:

struct Machine {}

struct ClosureContext<'a> {
    // `on_drop` must be dropped first
    on_drop: Vec<Box<dyn FnMut() + 'static>>,
    _machine: &'a Machine,
}

impl<'a> Drop for ClosureContext<'a> {
    fn drop(&mut self) {
        for f in &mut self.on_drop {
            f();
        }
    }
}

impl<'a> ClosureContext<'a> {
    fn push_closure<F>(&mut self, closure: F)
    where
        F: 'a + FnMut(),
    {
        let raw = Box::into_raw(Box::new(closure))
            as *mut std::os::raw::c_void;
        println!("Closure {:?} goes to C land", raw);
        self.on_drop.push(Box::new(move || {
            println!("Closure {:?} gets invalidated", raw);
        }));
    }
}

impl Machine {
    fn new_closure_context<'a, O>(&'a self, once: O)
    where
        for<'b> O: FnOnce(&'b mut ClosureContext<'a>),
    {
        let mut context = ClosureContext::<'a> {
            _machine: self,
            on_drop: vec![],
        };
        once(&mut context);
    }
    fn run(&self) {
        println!("Running (and possibly executing closures, unless invalidated)");
    }
}

fn main() {
    let machine = Machine {};
    let h: i32 = 0;
    let mut i: i32 = 0;
    let mut j: i32 = 0;
    machine.new_closure_context(|ctx| {
        ctx.push_closure(|| {
            i += 1;
        });
        ctx.push_closure(|| {
            j += 1;
        });

        // allowed:
        drop(h);

        // no effect, as ctx is just a mut reference:
        std::mem::forget(ctx);

        // not allowed:
        //drop(i);

        machine.run();
    });
    machine.run();
}

(Playground)

Output:

Closure 0x55affe0b69d0 goes to C land
Closure 0x55affe0b6a60 goes to C land
Running (and possibly executing closures, unless invalidated)
Closure 0x55affe0b69d0 gets invalidated
Closure 0x55affe0b6a60 gets invalidated
Running (and possibly executing closures, unless invalidated)

I had to use an HRTB, as you mentioned too. But I did not use any extra lifetime parameter to new_closure_context. Instead I just use 'a from &'a self and hope there is no problem with that?

I hope I (mostly) understood things right by not handing out ClosureContext and thus not allowing to mem::forget it. But I wouldn't be surprised if I got it all wrong :sweat_smile:.

Perhaps it's more clear to write it this way:

    fn new_closure_context<'a, 'b, O>(&'a self, once: O)
    where
        'a: 'b,
        for<'c> O: FnOnce(&'c mut ClosureContext<'b>),

This would indicate that the lifetime of the ClosureContext does not depend in any way on the lifetime of &self. But I believe in practice there is no difference to my approach where I used a single lifetime. Yet when I turn it into a trait, implementors of the trait will have to follow whichever signature I choose here, so not sure what's best to pick.

I'm still feeling confused to introduce 'b (or 'beyond_cleanup in your example) just to create a lifetime that lasts until the function returned. But it kinda makes sense. I think. :no_mouth:

Update:

I finally ended up with the following API interface (for now). Note that in my case, the Machine isn't a struct but a trait, because I intend to support an interface for multiple (different) scripting languages.

/// Ability of [`Machine`]s to call provided callbacks that live as long as the machine lives
pub trait Callback<'a>: Machine<'a> {
    /// Create a [`Machine::Datum`] representing a callback (which invokes the `func` closure)
    fn callback<F>(&'a self, func: F) -> Result<Self::Datum, MachineError>
    where
        F: 'a + FnMut(Vec<Self::Datum>) -> Result<Vec<Self::Datum>, String>;
}

/// Ability of [`Machine`]s to call provided callbacks that have a limited lifetime
pub trait ScopedCallback<'a>: Machine<'a> {
    /// Scope handle
    type Scope<'b>: CallbackScope<'a, 'b, Self>;
    /// Create scope handle that allows creation of callbacks with limited lifetime
    ///
    /// The scope handle will not be returned but a reference to the handle is
    /// passed to the given closure `once`.
    fn callback_scope<'b, O, R>(&'b self, once: O) -> R
    where
        O: FnOnce(&Self::Scope<'b>) -> R;
}

/// Scope that allows creation of callbacks with limited lifetime
pub trait CallbackScope<'a, 'b, M: Machine<'a> + ?Sized> {
    /// Create a [`Machine::Datum`] representing a callback (which invokes the `func` closure)
    ///
    /// After [`ScopedCallback::Scope`] has been dropped
    /// (i.e. after [`ScopedCallback::callback_scope`] returned),
    /// calling the callback will result in a runtime error ([`MachineError`]).
    fn callback<F>(&self, func: F) -> Result<M::Datum, MachineError>
    where
        F: 'b + FnMut(Vec<M::Datum>) -> Result<Vec<M::Datum>, String>;
}

Edit: Added lifetime 'a to self parameter in Callback::callback.

The associated type Machine::Datum is a value to be used in the scripting language, e.g. a Lua value such as a Lua string or Lua function (or closure).

  • The Callback trait allows converting a long-living Rust closure (i.e. a Rust closure that lives at least as long as the Machine) into a Machine::Datum.
  • The ScopedCallback trait allows creating a scope (some associated type which implements CallbackScope) which allows converting Rust closures into a Machine::Datum even if the Rust closures live for a short time only (i.e. if they are only valid until the once closure, which gets a reference to the scope handle, returns).

Note that I decided against using a dedicated lifetime parameter 'c, such as:

    fn callback_scope<'b, 'c, O, R>(&'b self, once: O) -> R
    where
        'b: 'c,
        O: FnOnce(&Self::Scope<'c>) -> R;

Using an extra 'c seems to make the interface more complex without any advantages (but please correct me if I'm wrong). Instead, I use a single lifetime 'b for both &'b self and &Self::Scope<'b>. Basically 'b can be any lifetime that is at least valid until callback_scope returns. If the reference to self actually lives longer, that should not be a problem, due to subtyping.

Also note that I ommitted the explicit HRTBs (to make the code a bit more compact). If I understand it right, they are still there (but implicit).

I will have to see if the interface is feasible. I have only implemented Callback yet, but not ScopedCallback. I might or might not face further problems when I actually try to provide implementations. Plus, the invalidation of an already created Lua closure (when the Rust closure and/or the scope handle goes out of scope) will add some complexity.

I'd be happy about some feedback regarding my API draft and I hope it's clear now how it's supposed to work.

11

I finally came up with another idea to solve this problem without using Drop glue, but instead using Rc<()>, Weak and lifetime transmutation :see_no_evil:.

I created a self-contained example that can be tested on Playground.

use std::cell::RefCell;
use std::marker::PhantomData;
use std::mem::transmute;
use std::rc::Rc;

struct Machine<'a> {
    callbacks: RefCell<Vec<Box<dyn 'a + FnMut()>>>,
}

type PhantomInvariant<T> = PhantomData<fn(T) -> T>;

struct CallbackScope<'a, 'b, 'c> {
    machine: &'b Machine<'a>,
    rc: Rc<()>,
    phantom: PhantomInvariant<&'c ()>,
}

impl<'a> Machine<'a> {
    fn new() -> Self {
        Machine {
            callbacks: RefCell::new(Vec::new()),
        }
    }
    fn add_callback<F>(&self, callback: F)
    where
        F: 'a + FnMut(),
    {
        self.callbacks.borrow_mut().push(Box::new(callback));
    }
    fn callback_scope<'b, 'c, O, R>(&'b self, once: O) -> R
    where
        O: FnOnce(&CallbackScope<'a, 'b, 'c>) -> R,
    {
        once(&CallbackScope::<'a, 'b, 'c> {
            machine: self,
            rc: Rc::new(()),
            phantom: PhantomData,
        })
    }
    fn run_callbacks(&self) {
        for callback in self.callbacks.borrow_mut().iter_mut() {
            callback();
        }
    }
}

impl<'a, 'b, 'c> CallbackScope<'a, 'b, 'c> {
    fn add_callback<F>(&self, func: F)
    where
        F: 'c + FnMut(),
    {
        let weak = Rc::downgrade(&self.rc);
        let boxed: Box<dyn 'c + FnMut()> = Box::new(func);
        let mut boxed: Box<dyn 'a + FnMut()> =
            unsafe { transmute(boxed) };
        self.machine.add_callback(move || match weak.upgrade() {
            Some(_) => boxed(),
            None => println!("(closure is no longer valid)"),
        })
    }
}

fn main() {
    let machine = Machine::new();
    let mut alice = 0;
    let mut bob = 0;
    machine.add_callback(|| {
        alice += 1;
        println!("Hello Alice!");
    });
    machine.run_callbacks();
    machine.callback_scope(|scope| {
        scope.add_callback(|| {
            bob += 1;
            println!("Hello Bob!");
        });
        machine.run_callbacks();
    });
    println!("Bob counter = {}", bob);
    drop(bob);
    machine.run_callbacks();
    drop(machine);
    println!("Alice counter = {}", alice);
    drop(alice);
}

(Playground)

Output:

Hello Alice!
Hello Alice!
Hello Bob!
Bob counter = 1
Hello Alice!
(closure is no longer valid)
Alice counter = 3

I'm not sure if it's correct already, but it seems to behave properly. (You can try to move the drops around.) Before you scold me for using unsafe without knowing what I'm doing: This is for me to learn more about the problem and how to solve it cleanly. I'm not going to use std::mem::transmute on lifetimes in productive code until I'm really understanding what's happening here. I'm still in the process of understanding what I have written there, and I'm still a bit puzzled about the PhantomInvariant and why I need it. If I use PhantomData<&'c ()> instead of PhantomInvariant<&'c ()>, then I can drop the bob counter while the once closure is executed, which is bad! I think that is because 'c can shrink, as @Yandros pointed out here:

But even if I followed that advice (I think), it's still very difficult for me to comprehend what I actually did. I'll certainly need more time to understand all this. Wanted to share my attempt anyway.

P.S.: I wonder if I can perform a lifetime transmutation of an opaque closure type without boxing it. Any way to do that?

12

I admit I've only skimmed this thread, but is there a real need for 'c on the CallbackScope itself, and choosable by the end-programmer? There's nothing inherently keeping it from being 'static, though it is bound based on the closures passed into CallbackScope::add_callback.

I.e. is there a downside to something like this, where the CallbackScope can only be ephemeral within Machine::callback_scope, and 'c is instead the lifetime of a borrow of the CallbackScope.

(I'm also somewhat guessing at things due to the lack of comments and privacy.)

Then I thought, "hmm it'd be nicer if expired callbacks got discarded", and arrived at this. In the process I realized that you're double-boxing your scoped closures, which you might want to avoid.

13

Thank you for your ideas on this, I will try to understand the implications of it once I have a better test scenario, because while testing your code and moving drops around, I figured out that drop won't actually drop the variables, as alice and bob are Copy.

Actually the drops don't really drop :pensive:.

fn main() {
    let i = 1;
    drop(i);
    drop(i);
    drop(i);
}

(Playground)

I remember I made this mistake before (and you even told me before).

In the most recent code, I tried to keep things simple.

My idea is that in the end, I have a trait that can be implemented by machines (e.g. by a Lua virtual machine) which allows converting long-living closures into a datum that's stored in the machine (e.g. a Lua closure). But I would like to provide a convenience wrapper which will also allow shorter-lived closures to be converted, resulting in a runtime-error if attempting to call the closure from Lua when it's no longer available.

I think in that case, the (transmuted) closure might be called even if it's no longer valid. Consider:

    /* … */
    machine.callback_scope(|scope| {
        {
            let mut danger = 0;
            scope.add_callback(|| {
                bob += 1;
                danger += 1;
                println!("Hello Bob!");
            });
            println!("dropping `danger`")
        }
        machine.run_callbacks();
    });
    println!("Bob counter = {}", bob);
    /* … */
}

(Playground)

Output:

Hello Alice!
dropping `danger`
Hello Alice!
Hello Bob!
Bob counter = 1
Hello Alice!
(closure is no longer valid)
Alice counter = 3

If I'm understanding it right, danger gets incremented after it's dropped. :see_no_evil:

I will re-evaluate my example and see if I can make it more sound and demonstrate better what I need/want. I think I'll have to invest some time into this, because closures could be a powerful mechanism to provide functionality within a sandbox that executes an interpreted language. Allowing short-lived closures to be injected as a Lua datum (technically it will be a C closure, using Lua's terminology) might make things much more easy for the user of such a sandbox on the Rust side.

Why do I want ephemeral closures at all?

Consider a case where some code in Rust wants to collect some data. It converts a Rust closure to a Lua datum and then calls some code in the Lua machine to gather the data (and use the closure to store the results). Eventually the collection process will be finished and the collected data is used by Rust. But the closure still exists in the Lua world. I cannot remotely destroy it there (well, maybe I could, but that might be tricky). If I used 'static with Arc or Rc, I would likely make the Rust closure return an error if being called too late. But instead of programming this "throw an error if called too late" functionality for each case, I want to include it in the virtual machine library and only program it once.

(Perhaps you could say "ephemeral closures" compare to normal closures as using RefCell/Ref/RefMut compares to using normal borrows. Instead of throwing compile-time errors, you'll get runtime errors.)

The problem shall be solved for more than one scripting language, i.e. the interface shall work with any scripting language (in theory). For some more info on my overall goals, you could check out this post and this thread if you like. But I'm still in the process of figuring out what I want/need, and I'm changing major parts of the API yet. That's why I tried to isolate the problem with my most recent post. Sorry if my references to Datum in my earlier posts caused confusion.

Edit: Explained drop problem better in beginning of post.

14

It looks like dealing with unsafe Rust, lifetimes, PhantomData, etc. is far more difficult as I expected. Some links in that matter:

I also stumbled upon NonZero, Unique, and Shared, as the Rustonomicon still mentions Unique in the section on Phantom Data. I searched for Unique :face_with_monocle: and didn't find it in the standard library. There is an old tracking issue #27730 on NonZero, Unique, Shared. The current standard library knows NonNull, which is covariant, but looks like Unique and Shared disappeared?

I feel like I have a lot to learn yet, even if I want to go the 'static way (so I really know what happens and when working with pointers is sound at all). There seem to be a lot of things that can go wrong whenever pointers are involved.

That said, I would like to learn it :grin:.

I believe that the idea of "ephemeral closures" or "weak closures" (not sure about terminology here) can be useful for a couple of usecases. Perhaps it would be best to implement this in a separate crate, which basically allows you to convert a closure with lifetime 'b into a closure with a longer lifetime 'a (or even 'static), whereas when the original closure (or the scope handle) goes out of scope, the closure with lifetime 'a will return an error (or execute a provided "error closure" with lifetime 'a).

If that has been solved, I could limit my own API to expect 'static closures or closures with a lifetime that is as long as the virutal machine exists. An "ephemeral closure" or "weak closure" crate would then allow to convert shorter living closures into longer living ones while specifying what shall happen in case the closure has been called too late (e.g. throw an error inside the VM).

15

This might be a nice generic tool to have; something that lets you do RefCell-like runtime borrow checking but applying to normal references (when used by functions) rather than a distinct type. Note that it still has to use the same callback_scope(|ctx| {...}) style mechanism to establish the scope; this would just make it separate from the types it's being used with.

Caveat: Rust code currently can't be generic over function arity, which makes it less convenient (or more work for this hypothetical generic library) if the callback you want to "weaken" is expected to have more than one.

1 Like
16

Yeah, you're right. (Change danger to a String and print it... or rather, print a dropped String :skull_and_crossbones: .)

17

I continued discussing this idea in the thread Ephemeral (or “weak”) closures.

Another thing I stumbled upon yesterday is that if I pass Rust closures to a Lua machine, I will need to make sure the Lua machine is !Send, because otherwise the closures might hold an immutable reference to a !Sync value (and be executed in a different thread).

Naturally, when I include a raw pointer in a struct, that makes the whole struct !Send and !Sync, I think. Thus I assume LuaMachine<'a> is !Send if I declare it like this:

/// Virtual machine that executes Lua 5.4 code
#[derive(Debug)]
pub struct LuaMachine<'a> {
    c_machine: *mut cmach_lua_t,
    phantom: PhantomData<&'a ()>,
}

So I should be on the safe side here. I have read that a lot of code actually relies on that. See also discussion Shouldn’t pointers be Send + Sync? Or.

Originally, I wanted to add something like the following to my code, which is dangerous if there are closures around which are !Send, which are then executed by the Lua machine:

/// Allow sending machine to other threads
/// (any loaded libraries must cope with that)
unsafe impl<'a> Send for LuaMachine<'a> {}
18

Am I correct that VirtualMachine<'a> needs to be invariant (or contravariant, but not covariant) in 'a?

Consider:

use std::marker::PhantomData;

struct VirtualMachine<'a> {
    _phantom: PhantomData<&'a ()>, // DANGER, we're covariant in 'a!
}

impl<'a> VirtualMachine<'a> {
    fn new() -> Self {
        VirtualMachine { _phantom: PhantomData }
    }
    fn add_closure<F: 'a + FnMut()>(&self, _func: F) {
        // let's assume `_func` gets processed by C code from here on
    }
    fn use_closures(&self) {
        // let's assume the added closure(s) get called by C code
    }
}

fn main() {
    let m = VirtualMachine::new();
    {
        let s = "Hello".to_string();
        m.add_closure(|| println!("{}", &s));
    }
    m.use_closures(); // DANGER!
}

(Playground) (edited to make println! work explicitly on &s to emphasize the problem)

While one might be tempted to solve this by making add_closure work on &mut self (thus avoiding inner mutability), I'm not sure if that would entirely solve the problem, due to what @steffahn wrote in the other thread on Ephemeral (or Weak) Closures:

19

Yes. Your type is effectively “a function that takes &'a Ts” and so it must be contravariant in 'a to avoid the unsound lifetime shortening you described. It might also need to be invariant if the closures can be taken out again, just like any mutable container is invariant in the type of its contents.

1 Like
20

Do you (or anyone else) know whether I also need contravariance/invariance when I use &mut self? A type like Vec<T> (no inner mutability) is covariant in T. I'm just confused/cautious because &mut self didn't help in the case of a scope for the ephemeral closure experiment (due to replace_with::replace_with_or_abort or similar APIs that are considered to be safe).