Passing Rust closures to C

This is not sound because callers can leak the guard, preventing its drop code from running. To enforce cleanup at a stack frame you have to instead write a function that takes a closure and immediately calls it — then the function is guaranteed to see a return or panic.

I would summarize it like so: In &'a mut SomeType<'b>, the implicit lifetime requirements are that 'b must be longer than (or equal to) the existence of this instance of SomeType, and 'a must be shorter (or equal). Both of these correspond to the direction of the references they're protecting: a reference must be outlived by its referent.

1 Like

@Yandros: Wanted to say thanks already for your detailed post. I'm still digesting/processing it.

That makes sense. When you don't get ownership of the handle, you can't mem::forget it.

… okay, that's unexpected, but very interesting.

1 Like

I'm still having a hard time following through your post. I haven't worked with scopeguard yet, but I understand what it's supposed to do. But where I can't follow is what a PhantomInvariant is, for example.

I also don't understand why there is an impl Drop within a function body in the "Final signature".

Anyway, I tried to put those pieces of your post which I did understand together, and this is what I came up with:

struct Machine {}

struct ClosureContext<'a> {
    // `on_drop` must be dropped first
    on_drop: Vec<Box<dyn FnMut() + 'static>>,
    _machine: &'a Machine,
}

impl<'a> Drop for ClosureContext<'a> {
    fn drop(&mut self) {
        for f in &mut self.on_drop {
            f();
        }
    }
}

impl<'a> ClosureContext<'a> {
    fn push_closure<F>(&mut self, closure: F)
    where
        F: 'a + FnMut(),
    {
        let raw = Box::into_raw(Box::new(closure))
            as *mut std::os::raw::c_void;
        println!("Closure {:?} goes to C land", raw);
        self.on_drop.push(Box::new(move || {
            println!("Closure {:?} gets invalidated", raw);
        }));
    }
}

impl Machine {
    fn new_closure_context<'a, O>(&'a self, once: O)
    where
        for<'b> O: FnOnce(&'b mut ClosureContext<'a>),
    {
        let mut context = ClosureContext::<'a> {
            _machine: self,
            on_drop: vec![],
        };
        once(&mut context);
    }
    fn run(&self) {
        println!("Running (and possibly executing closures, unless invalidated)");
    }
}

fn main() {
    let machine = Machine {};
    let h: i32 = 0;
    let mut i: i32 = 0;
    let mut j: i32 = 0;
    machine.new_closure_context(|ctx| {
        ctx.push_closure(|| {
            i += 1;
        });
        ctx.push_closure(|| {
            j += 1;
        });

        // allowed:
        drop(h);

        // no effect, as ctx is just a mut reference:
        std::mem::forget(ctx);

        // not allowed:
        //drop(i);

        machine.run();
    });
    machine.run();
}

(Playground)

Output:

Closure 0x55affe0b69d0 goes to C land
Closure 0x55affe0b6a60 goes to C land
Running (and possibly executing closures, unless invalidated)
Closure 0x55affe0b69d0 gets invalidated
Closure 0x55affe0b6a60 gets invalidated
Running (and possibly executing closures, unless invalidated)

I had to use an HRTB, as you mentioned too. But I did not use any extra lifetime parameter to new_closure_context. Instead I just use 'a from &'a self and hope there is no problem with that?

I hope I (mostly) understood things right by not handing out ClosureContext and thus not allowing to mem::forget it. But I wouldn't be surprised if I got it all wrong :sweat_smile:.


Perhaps it's more clear to write it this way:

    fn new_closure_context<'a, 'b, O>(&'a self, once: O)
    where
        'a: 'b,
        for<'c> O: FnOnce(&'c mut ClosureContext<'b>),

This would indicate that the lifetime of the ClosureContext does not depend in any way on the lifetime of &self. But I believe in practice there is no difference to my approach where I used a single lifetime. Yet when I turn it into a trait, implementors of the trait will have to follow whichever signature I choose here, so not sure what's best to pick.

I'm still feeling confused to introduce 'b (or 'beyond_cleanup in your example) just to create a lifetime that lasts until the function returned. But it kinda makes sense. I think. :no_mouth:


Update:

I finally ended up with the following API interface (for now). Note that in my case, the Machine isn't a struct but a trait, because I intend to support an interface for multiple (different) scripting languages.

/// Ability of [`Machine`]s to call provided callbacks that live as long as the machine lives
pub trait Callback<'a>: Machine<'a> {
    /// Create a [`Machine::Datum`] representing a callback (which invokes the `func` closure)
    fn callback<F>(&'a self, func: F) -> Result<Self::Datum, MachineError>
    where
        F: 'a + FnMut(Vec<Self::Datum>) -> Result<Vec<Self::Datum>, String>;
}

/// Ability of [`Machine`]s to call provided callbacks that have a limited lifetime
pub trait ScopedCallback<'a>: Machine<'a> {
    /// Scope handle
    type Scope<'b>: CallbackScope<'a, 'b, Self>;
    /// Create scope handle that allows creation of callbacks with limited lifetime
    ///
    /// The scope handle will not be returned but a reference to the handle is
    /// passed to the given closure `once`.
    fn callback_scope<'b, O, R>(&'b self, once: O) -> R
    where
        O: FnOnce(&Self::Scope<'b>) -> R;
}

/// Scope that allows creation of callbacks with limited lifetime
pub trait CallbackScope<'a, 'b, M: Machine<'a> + ?Sized> {
    /// Create a [`Machine::Datum`] representing a callback (which invokes the `func` closure)
    ///
    /// After [`ScopedCallback::Scope`] has been dropped
    /// (i.e. after [`ScopedCallback::callback_scope`] returned),
    /// calling the callback will result in a runtime error ([`MachineError`]).
    fn callback<F>(&self, func: F) -> Result<M::Datum, MachineError>
    where
        F: 'b + FnMut(Vec<M::Datum>) -> Result<Vec<M::Datum>, String>;
}

Edit: Added lifetime 'a to self parameter in Callback::callback.

The associated type Machine::Datum is a value to be used in the scripting language, e.g. a Lua value such as a Lua string or Lua function (or closure).

  • The Callback trait allows converting a long-living Rust closure (i.e. a Rust closure that lives at least as long as the Machine) into a Machine::Datum.
  • The ScopedCallback trait allows creating a scope (some associated type which implements CallbackScope) which allows converting Rust closures into a Machine::Datum even if the Rust closures live for a short time only (i.e. if they are only valid until the once closure, which gets a reference to the scope handle, returns).

Note that I decided against using a dedicated lifetime parameter 'c, such as:

    fn callback_scope<'b, 'c, O, R>(&'b self, once: O) -> R
    where
        'b: 'c,
        O: FnOnce(&Self::Scope<'c>) -> R;

Using an extra 'c seems to make the interface more complex without any advantages (but please correct me if I'm wrong). Instead, I use a single lifetime 'b for both &'b self and &Self::Scope<'b>. Basically 'b can be any lifetime that is at least valid until callback_scope returns. If the reference to self actually lives longer, that should not be a problem, due to subtyping.

Also note that I ommitted the explicit HRTBs (to make the code a bit more compact). If I understand it right, they are still there (but implicit).

I will have to see if the interface is feasible. I have only implemented Callback yet, but not ScopedCallback. I might or might not face further problems when I actually try to provide implementations. Plus, the invalidation of an already created Lua closure (when the Rust closure and/or the scope handle goes out of scope) will add some complexity.

I'd be happy about some feedback regarding my API draft and I hope it's clear now how it's supposed to work.

I finally came up with another idea to solve this problem without using Drop glue, but instead using Rc<()>, Weak and lifetime transmutation :see_no_evil:.

I created a self-contained example that can be tested on Playground.

use std::cell::RefCell;
use std::marker::PhantomData;
use std::mem::transmute;
use std::rc::Rc;

struct Machine<'a> {
    callbacks: RefCell<Vec<Box<dyn 'a + FnMut()>>>,
}

type PhantomInvariant<T> = PhantomData<fn(T) -> T>;

struct CallbackScope<'a, 'b, 'c> {
    machine: &'b Machine<'a>,
    rc: Rc<()>,
    phantom: PhantomInvariant<&'c ()>,
}

impl<'a> Machine<'a> {
    fn new() -> Self {
        Machine {
            callbacks: RefCell::new(Vec::new()),
        }
    }
    fn add_callback<F>(&self, callback: F)
    where
        F: 'a + FnMut(),
    {
        self.callbacks.borrow_mut().push(Box::new(callback));
    }
    fn callback_scope<'b, 'c, O, R>(&'b self, once: O) -> R
    where
        O: FnOnce(&CallbackScope<'a, 'b, 'c>) -> R,
    {
        once(&CallbackScope::<'a, 'b, 'c> {
            machine: self,
            rc: Rc::new(()),
            phantom: PhantomData,
        })
    }
    fn run_callbacks(&self) {
        for callback in self.callbacks.borrow_mut().iter_mut() {
            callback();
        }
    }
}

impl<'a, 'b, 'c> CallbackScope<'a, 'b, 'c> {
    fn add_callback<F>(&self, func: F)
    where
        F: 'c + FnMut(),
    {
        let weak = Rc::downgrade(&self.rc);
        let boxed: Box<dyn 'c + FnMut()> = Box::new(func);
        let mut boxed: Box<dyn 'a + FnMut()> =
            unsafe { transmute(boxed) };
        self.machine.add_callback(move || match weak.upgrade() {
            Some(_) => boxed(),
            None => println!("(closure is no longer valid)"),
        })
    }
}

fn main() {
    let machine = Machine::new();
    let mut alice = 0;
    let mut bob = 0;
    machine.add_callback(|| {
        alice += 1;
        println!("Hello Alice!");
    });
    machine.run_callbacks();
    machine.callback_scope(|scope| {
        scope.add_callback(|| {
            bob += 1;
            println!("Hello Bob!");
        });
        machine.run_callbacks();
    });
    println!("Bob counter = {}", bob);
    drop(bob);
    machine.run_callbacks();
    drop(machine);
    println!("Alice counter = {}", alice);
    drop(alice);
}

(Playground)

Output:

Hello Alice!
Hello Alice!
Hello Bob!
Bob counter = 1
Hello Alice!
(closure is no longer valid)
Alice counter = 3

I'm not sure if it's correct already, but it seems to behave properly. (You can try to move the drops around.) Before you scold me for using unsafe without knowing what I'm doing: This is for me to learn more about the problem and how to solve it cleanly. I'm not going to use std::mem::transmute on lifetimes in productive code until I'm really understanding what's happening here. I'm still in the process of understanding what I have written there, and I'm still a bit puzzled about the PhantomInvariant and why I need it. If I use PhantomData<&'c ()> instead of PhantomInvariant<&'c ()>, then I can drop the bob counter while the once closure is executed, which is bad! I think that is because 'c can shrink, as @Yandros pointed out here:

But even if I followed that advice (I think), it's still very difficult for me to comprehend what I actually did. I'll certainly need more time to understand all this. Wanted to share my attempt anyway.

P.S.: I wonder if I can perform a lifetime transmutation of an opaque closure type without boxing it. Any way to do that?

I admit I've only skimmed this thread, but is there a real need for 'c on the CallbackScope itself, and choosable by the end-programmer? There's nothing inherently keeping it from being 'static, though it is bound based on the closures passed into CallbackScope::add_callback.

I.e. is there a downside to something like this, where the CallbackScope can only be ephemeral within Machine::callback_scope, and 'c is instead the lifetime of a borrow of the CallbackScope.

(I'm also somewhat guessing at things due to the lack of comments and privacy.)


Then I thought, "hmm it'd be nicer if expired callbacks got discarded", and arrived at this. In the process I realized that you're double-boxing your scoped closures, which you might want to avoid.

Thank you for your ideas on this, I will try to understand the implications of it once I have a better test scenario, because while testing your code and moving drops around, I figured out that drop won't actually drop the variables, as alice and bob are Copy.

Actually the drops don't really drop :pensive:.

fn main() {
    let i = 1;
    drop(i);
    drop(i);
    drop(i);
}

(Playground)

I remember I made this mistake before (and you even told me before).

In the most recent code, I tried to keep things simple.

My idea is that in the end, I have a trait that can be implemented by machines (e.g. by a Lua virtual machine) which allows converting long-living closures into a datum that's stored in the machine (e.g. a Lua closure). But I would like to provide a convenience wrapper which will also allow shorter-lived closures to be converted, resulting in a runtime-error if attempting to call the closure from Lua when it's no longer available.

I think in that case, the (transmuted) closure might be called even if it's no longer valid. Consider:

    /* … */
    machine.callback_scope(|scope| {
        {
            let mut danger = 0;
            scope.add_callback(|| {
                bob += 1;
                danger += 1;
                println!("Hello Bob!");
            });
            println!("dropping `danger`")
        }
        machine.run_callbacks();
    });
    println!("Bob counter = {}", bob);
    /* … */
}

(Playground)

Output:

Hello Alice!
dropping `danger`
Hello Alice!
Hello Bob!
Bob counter = 1
Hello Alice!
(closure is no longer valid)
Alice counter = 3

If I'm understanding it right, danger gets incremented after it's dropped. :see_no_evil:

I will re-evaluate my example and see if I can make it more sound and demonstrate better what I need/want. I think I'll have to invest some time into this, because closures could be a powerful mechanism to provide functionality within a sandbox that executes an interpreted language. Allowing short-lived closures to be injected as a Lua datum (technically it will be a C closure, using Lua's terminology) might make things much more easy for the user of such a sandbox on the Rust side.

Why do I want ephemeral closures at all?

Consider a case where some code in Rust wants to collect some data. It converts a Rust closure to a Lua datum and then calls some code in the Lua machine to gather the data (and use the closure to store the results). Eventually the collection process will be finished and the collected data is used by Rust. But the closure still exists in the Lua world. I cannot remotely destroy it there (well, maybe I could, but that might be tricky). If I used 'static with Arc or Rc, I would likely make the Rust closure return an error if being called too late. But instead of programming this "throw an error if called too late" functionality for each case, I want to include it in the virtual machine library and only program it once.

(Perhaps you could say "ephemeral closures" compare to normal closures as using RefCell/Ref/RefMut compares to using normal borrows. Instead of throwing compile-time errors, you'll get runtime errors.)

The problem shall be solved for more than one scripting language, i.e. the interface shall work with any scripting language (in theory). For some more info on my overall goals, you could check out this post and this thread if you like. But I'm still in the process of figuring out what I want/need, and I'm changing major parts of the API yet. That's why I tried to isolate the problem with my most recent post. Sorry if my references to Datum in my earlier posts caused confusion.

Edit: Explained drop problem better in beginning of post.

It looks like dealing with unsafe Rust, lifetimes, PhantomData, etc. is far more difficult as I expected. Some links in that matter:

I also stumbled upon NonZero, Unique, and Shared, as the Rustonomicon still mentions Unique in the section on Phantom Data. I searched for Unique :face_with_monocle: and didn't find it in the standard library. There is an old tracking issue #27730 on NonZero, Unique, Shared. The current standard library knows NonNull, which is covariant, but looks like Unique and Shared disappeared?

I feel like I have a lot to learn yet, even if I want to go the 'static way (so I really know what happens and when working with pointers is sound at all). There seem to be a lot of things that can go wrong whenever pointers are involved.

That said, I would like to learn it :grin:.

I believe that the idea of "ephemeral closures" or "weak closures" (not sure about terminology here) can be useful for a couple of usecases. Perhaps it would be best to implement this in a separate crate, which basically allows you to convert a closure with lifetime 'b into a closure with a longer lifetime 'a (or even 'static), whereas when the original closure (or the scope handle) goes out of scope, the closure with lifetime 'a will return an error (or execute a provided "error closure" with lifetime 'a).

If that has been solved, I could limit my own API to expect 'static closures or closures with a lifetime that is as long as the virutal machine exists. An "ephemeral closure" or "weak closure" crate would then allow to convert shorter living closures into longer living ones while specifying what shall happen in case the closure has been called too late (e.g. throw an error inside the VM).

This might be a nice generic tool to have; something that lets you do RefCell-like runtime borrow checking but applying to normal references (when used by functions) rather than a distinct type. Note that it still has to use the same callback_scope(|ctx| {...}) style mechanism to establish the scope; this would just make it separate from the types it's being used with.

Caveat: Rust code currently can't be generic over function arity, which makes it less convenient (or more work for this hypothetical generic library) if the callback you want to "weaken" is expected to have more than one.

1 Like

Yeah, you're right. (Change danger to a String and print it... or rather, print a dropped String :skull_and_crossbones: .)

I continued discussing this idea in the thread Ephemeral (or “weak”) closures.

Another thing I stumbled upon yesterday is that if I pass Rust closures to a Lua machine, I will need to make sure the Lua machine is !Send, because otherwise the closures might hold an immutable reference to a !Sync value (and be executed in a different thread).

Naturally, when I include a raw pointer in a struct, that makes the whole struct !Send and !Sync, I think. Thus I assume LuaMachine<'a> is !Send if I declare it like this:

/// Virtual machine that executes Lua 5.4 code
#[derive(Debug)]
pub struct LuaMachine<'a> {
    c_machine: *mut cmach_lua_t,
    phantom: PhantomData<&'a ()>,
}

So I should be on the safe side here. I have read that a lot of code actually relies on that. See also discussion Shouldn’t pointers be Send + Sync? Or.

Originally, I wanted to add something like the following to my code, which is dangerous if there are closures around which are !Send, which are then executed by the Lua machine:

/// Allow sending machine to other threads
/// (any loaded libraries must cope with that)
unsafe impl<'a> Send for LuaMachine<'a> {}

Am I correct that VirtualMachine<'a> needs to be invariant (or contravariant, but not covariant) in 'a?

Consider:

use std::marker::PhantomData;

struct VirtualMachine<'a> {
    _phantom: PhantomData<&'a ()>, // DANGER, we're covariant in 'a!
}

impl<'a> VirtualMachine<'a> {
    fn new() -> Self {
        VirtualMachine { _phantom: PhantomData }
    }
    fn add_closure<F: 'a + FnMut()>(&self, _func: F) {
        // let's assume `_func` gets processed by C code from here on
    }
    fn use_closures(&self) {
        // let's assume the added closure(s) get called by C code
    }
}

fn main() {
    let m = VirtualMachine::new();
    {
        let s = "Hello".to_string();
        m.add_closure(|| println!("{}", &s));
    }
    m.use_closures(); // DANGER!
}

(Playground) (edited to make println! work explicitly on &s to emphasize the problem)

While one might be tempted to solve this by making add_closure work on &mut self (thus avoiding inner mutability), I'm not sure if that would entirely solve the problem, due to what @steffahn wrote in the other thread on Ephemeral (or Weak) Closures:

Yes. Your type is effectively “a function that takes &'a Ts” and so it must be contravariant in 'a to avoid the unsound lifetime shortening you described. It might also need to be invariant if the closures can be taken out again, just like any mutable container is invariant in the type of its contents.

1 Like

Do you (or anyone else) know whether I also need contravariance/invariance when I use &mut self? A type like Vec<T> (no inner mutability) is covariant in T. I'm just confused/cautious because &mut self didn't help in the case of a scope for the ephemeral closure experiment (due to replace_with::replace_with_or_abort or similar APIs that are considered to be safe).

&mut T is invariant in T (including when T = Self). I didn't follow everything you've tried so I can't say why it would be insufficient, but maybe it has to do with the owned value continuing to exist after a particular mutable borrow of it is dropped (and thus the owned value could expose inappropriate covariance even when the mutable reference does not)?

Not sure if I understand what you mean with "the owned value continuing to exist", and I understand you haven't followed the other thread (it got quite long :sweat_smile:).

I'll try to summarize my understanding anyway for now.

  • As @steffahn showed here, when having a mutable reference to a scope handle with a 'scope lifetime parameter, it's possible to obtain an owned scope handle with that lifetime (temporariliy, before panicking) using replace_with::replace_with_or_abort. Now if the handle is covariant in 'scope, you can write let mut handle = handle; to shrink the lifetime parameter of the handle. If the handle now offers converting the closure with such shrunk lifetime (normally 'scope, but now an even shorter lifetime) to a closure with a longer lifetime (e.g. 'static) under the assumption that the handle will be dropped when the lifetime of the original closure ends, then you can exploit this by mem::forgetting the scope handle such that it's destructors aren't run. I think the problem here is that the returned closure continues to exist (and can be used) even if the scope handle is gone (forgotten).

Now back to this thread:

  • Under the assumption that I require a mutable reference to add closures to a virtual machine and a (mutable or immutable) reference to execute those closures, I think it would be sufficient if the virtual machine has a lifetime parameter that is covariant (akin to to Vec<T>, which is also covariant in T). But the important prerequisite here is that I can execute the closures only as long as the machine still exists (as in I can provide a reference to it; that is a mutable reference when executing a closure that's FnMut).

I still have to decide whether I use inner mutability (and provide an API that takes immutable references to the machine for adding closures), or make the API require mutable references to the VM for adding closures or for executing FnMuts. The latter case might make things quite complex for the user, but perhaps it's the cleaner approach. I would assume that when going the clean approach, it would be sufficient if the virtual machine was covariant in the lifetime parameter (like a Vec<T> is covariant in T).

But not sure.

(I know it is difficult to follow without the particular code. Right now my project is messed up as making the machine invariant in the lifetime parameter broke everything :sweat_smile:, which actually means all my code was flawed before. I'll have to rebuild things from the beginning, so that will take time.)

Anyway, thanks already for all the help!

This is what I was thinking of:

/// 'a is covariant per usual expectations
struct HasCovariantLifetime<'a> {
    contents: &'a str,
}

/// 'a is invariant because it appears inside an &mut
fn borrow_hcl<'a, 'r>(hcl: &'r mut HasCovariantLifetime<'a>) {}

fn main() {
    let mut hcl = HasCovariantLifetime { contents: "" };
    borrow_hcl(&mut hcl);
    borrow_hcl(&mut hcl);  // This compiles
}

borrow_hcl is invariant in its 'a, but HasCovariantLifetime is covariant in its 'a. So, borrow_hcl itself experiences the restriction of invariance, but cannot soundly assume that future uses of HasCovariantLifetime won't have a shorter lifetime for 'a. In particular,

fn borrow_hcl<'a, 'r>(hcl: &'r mut HasCovariantLifetime<'a>, input: &'a str) {
    hcl.contents = input;
}

fn main() {
    let mut hcl = HasCovariantLifetime { contents: "" };
    {  
        let s1 = String::from("foo");
        borrow_hcl(&mut hcl, &s1);
        {  
            let s2 = String::from("bar");
            borrow_hcl(&mut hcl, &s2);
        }
    }
}

hcl's lifetime parameter at the end must be shorter than or equal to s2's existence, but it was also used previously with s1 when the lifetime of s2 hadn't started yet, so this exhibits covariance in combination with &mut. That said, I'm not sure how to relate this back to the unsoundness problem and I've spent a fair bit of time trying to think about it but not gotten anywhere good. I hope I explained what I meant, at least.

1 Like

Yes, think I understand now. Thank you for elaborating. I don't think it's the same as the problem that occurred in the other thread. There, the problem wasn't a future use of the handle but the use of a transmuted result contained in the return value from a previous use (I think).

In your example, when you borrow hcl later, there will be an error:

fn main() {
    let mut hcl = HasCovariantLifetime { contents: "" };
    {  
        let s1 = String::from("foo");
        borrow_hcl(&mut hcl, &s1);
        {  
            let s2 = String::from("bar");
            borrow_hcl(&mut hcl, &s2);
        }
+       borrow_hcl(&mut hcl, &s1);
    }
}

(Playground)

The same happens if you try to get an immutable reference (Playground) or drop the value (Playground) later.

So I think it's safe if the VM is covariant over a lifetime parameter 'a under the preconditions that:

  • storing closures (of lifetime 'a) will require a mutable reference to the VM,
  • calling closures will require a reference (mutable when calling stored FnMut's, or or immutable when calling stored Fn's) to the VM.

I think that invariance is needed when storing the closures can be done with an immutable reference to the VM (just like PhantomData<Cell<T>> should be invariant over T, I assume).

However, all of this is still very confusing to me, and I will need a lot of time yet to better understand what happens, especially with the borrow checker. Maybe going over simpler examples over and over again might do the trick for me :sweat_smile:. (But also happy if someone has a good link to a tutorial to better understand borrowing, variance, covariance, reborrowing, etc.)

I've only lightly skimmed (the last few posts of) the discussion in this thread, but the examples here don't involve any scope-style functions that only introduce a handle by-reference in a call-back (in order to guarantee that all handles are dropped (before their lifetime parameter ends)). The unsoundness of a covariant handle on the other thread was based on the same potential soundness issue that prevented you from having an ordinary constructor in the first place (instead of that scope function expecting a callback).

TL;DR, I see an ordinary constructor here, no function with callbacks, so the situation is significantly different.

1 Like

I assume the fact that the example worked at all is also due to non-lexical lifetimes.

Thanks!

I'm currently tempted to make any calls that run code in the VM work on &mut machine. That way, I can also (safely) invoke FnMut's from the scripting language. Under these assumptions, I believe the Machine can be covariant over the lifetime parameter (just like a Vec<T> is covariant over T because you can only modify it when having a mutable reference). But I'm still experimenting.


Update: Apparently making such a VM operate on &mut self causes some trouble. Consider a simple case such as:

struct Machine {
    s: String,
}

impl Machine {
    fn getstr(&mut self) -> &str {
        &self.s
    }
}

fn main() {
    let mut m = Machine {
        s: "Hello".to_string(),
    };
    let _s1 = m.getstr();
    let _s2 = m.getstr();
    // Uncommenting the following line is an error:
    //println!("{} {}", _s1, _s2);
}

(Playground)

That's one of the setbacks I was talking about here :sweat_smile:. I had to write down this tiny example to be sure how things really are.

I can fix that by using a lot of Rc and Weak, etc., but possibly using interior mutability (i.e. making all operations on the VM require only an immutable/shared reference) makes things easier, because I can return values with a reasonable lifetime (that lasts while the machine is existent). And then I need invariance.

But I'm still unsure which variant is best. The whole task seems to be pretty difficult to judge about, maybe because I'm attempting to connect a very strict language with lifetimes (Rust) with a language that is highly dynamic and uses garbage collection (Lua). It feels like a real mismatch, but I'm determined to find a solution that works nicely.

I followed some of the advice here:

So I went ahead, but I ran into some annoying issues when demanding invariance. Let me provide a boiled-down example:

use std::marker::PhantomData;

#[derive(Default)]
struct Machine<'a> {
    _phantom: PhantomData<fn(&'a ()) -> &'a ()>,
}

struct Value<'a> {
    _machine: &'a Machine<'a>,
}

impl<'a> Machine<'a> {
    fn get_value(&'a self) -> Value<'a> {
        Value { _machine: self }
    }
}

// I cannot uncomment this:
/*
impl<'a> Drop for Machine<'a> {
    fn drop(&mut self) {}
}
*/

fn main() {
    let m: Machine = Default::default();
    let v = m.get_value();
    drop(v);
    // And I cannot uncomment this:
    //drop(m);
}

(Playground)

I think this is related to having the same lifetime ('a) in the reference and lifetime argument in _machine: &'a Machine<'a>, and the fact that the borrow of the Machine has a slighly shorter lifetime than the 'a of the lifetime argument.

I can fix this by using two lifetimes:

-struct Value<'a> {
+struct Value<'a, 'b> {
-    _machine: &'a Machine<'a>,
+    _machine: &'b Machine<'a>,
 }
 
 impl<'a> Machine<'a> {
-    fn get_value(&'a self) -> Value<'a> {
+    fn get_value<'b>(&'b self) -> Value<'a, 'b> {
         Value { _machine: self }
     }
 }

But this feels a bit annoying because the lifetimes basically will be almost the same in all cases.

I could also fix this by using my favorite technique of lifetime transmutation :innocent: :innocent: :innocent: (and shadowing the owned Machine):

 fn main() {
     let m: Machine = Default::default();
+    let m = unsafe { std::mem::transmute::<&Machine, &Machine>(&m) };

But I don't want to do that in productive code, of course.

So is there any clean(er) way to work around my problem of introducing an extra lifetime 'b that is basically (almost) the same as 'a? It would mean changing dozens of data types to accept an extra lifetime.