Passing Rust closures to C

&mut T is invariant in T (including when T = Self). I didn't follow everything you've tried so I can't say why it would be insufficient, but maybe it has to do with the owned value continuing to exist after a particular mutable borrow of it is dropped (and thus the owned value could expose inappropriate covariance even when the mutable reference does not)?

Not sure if I understand what you mean with "the owned value continuing to exist", and I understand you haven't followed the other thread (it got quite long :sweat_smile:).

I'll try to summarize my understanding anyway for now.

  • As @steffahn showed here, when having a mutable reference to a scope handle with a 'scope lifetime parameter, it's possible to obtain an owned scope handle with that lifetime (temporariliy, before panicking) using replace_with::replace_with_or_abort. Now if the handle is covariant in 'scope, you can write let mut handle = handle; to shrink the lifetime parameter of the handle. If the handle now offers converting the closure with such shrunk lifetime (normally 'scope, but now an even shorter lifetime) to a closure with a longer lifetime (e.g. 'static) under the assumption that the handle will be dropped when the lifetime of the original closure ends, then you can exploit this by mem::forgetting the scope handle such that it's destructors aren't run. I think the problem here is that the returned closure continues to exist (and can be used) even if the scope handle is gone (forgotten).

Now back to this thread:

  • Under the assumption that I require a mutable reference to add closures to a virtual machine and a (mutable or immutable) reference to execute those closures, I think it would be sufficient if the virtual machine has a lifetime parameter that is covariant (akin to to Vec<T>, which is also covariant in T). But the important prerequisite here is that I can execute the closures only as long as the machine still exists (as in I can provide a reference to it; that is a mutable reference when executing a closure that's FnMut).

I still have to decide whether I use inner mutability (and provide an API that takes immutable references to the machine for adding closures), or make the API require mutable references to the VM for adding closures or for executing FnMuts. The latter case might make things quite complex for the user, but perhaps it's the cleaner approach. I would assume that when going the clean approach, it would be sufficient if the virtual machine was covariant in the lifetime parameter (like a Vec<T> is covariant in T).

But not sure.

(I know it is difficult to follow without the particular code. Right now my project is messed up as making the machine invariant in the lifetime parameter broke everything :sweat_smile:, which actually means all my code was flawed before. I'll have to rebuild things from the beginning, so that will take time.)

Anyway, thanks already for all the help!

This is what I was thinking of:

/// 'a is covariant per usual expectations
struct HasCovariantLifetime<'a> {
    contents: &'a str,
}

/// 'a is invariant because it appears inside an &mut
fn borrow_hcl<'a, 'r>(hcl: &'r mut HasCovariantLifetime<'a>) {}

fn main() {
    let mut hcl = HasCovariantLifetime { contents: "" };
    borrow_hcl(&mut hcl);
    borrow_hcl(&mut hcl);  // This compiles
}

borrow_hcl is invariant in its 'a, but HasCovariantLifetime is covariant in its 'a. So, borrow_hcl itself experiences the restriction of invariance, but cannot soundly assume that future uses of HasCovariantLifetime won't have a shorter lifetime for 'a. In particular,

fn borrow_hcl<'a, 'r>(hcl: &'r mut HasCovariantLifetime<'a>, input: &'a str) {
    hcl.contents = input;
}

fn main() {
    let mut hcl = HasCovariantLifetime { contents: "" };
    {  
        let s1 = String::from("foo");
        borrow_hcl(&mut hcl, &s1);
        {  
            let s2 = String::from("bar");
            borrow_hcl(&mut hcl, &s2);
        }
    }
}

hcl's lifetime parameter at the end must be shorter than or equal to s2's existence, but it was also used previously with s1 when the lifetime of s2 hadn't started yet, so this exhibits covariance in combination with &mut. That said, I'm not sure how to relate this back to the unsoundness problem and I've spent a fair bit of time trying to think about it but not gotten anywhere good. I hope I explained what I meant, at least.

1 Like

Yes, think I understand now. Thank you for elaborating. I don't think it's the same as the problem that occurred in the other thread. There, the problem wasn't a future use of the handle but the use of a transmuted result contained in the return value from a previous use (I think).

In your example, when you borrow hcl later, there will be an error:

fn main() {
    let mut hcl = HasCovariantLifetime { contents: "" };
    {  
        let s1 = String::from("foo");
        borrow_hcl(&mut hcl, &s1);
        {  
            let s2 = String::from("bar");
            borrow_hcl(&mut hcl, &s2);
        }
+       borrow_hcl(&mut hcl, &s1);
    }
}

(Playground)

The same happens if you try to get an immutable reference (Playground) or drop the value (Playground) later.

So I think it's safe if the VM is covariant over a lifetime parameter 'a under the preconditions that:

  • storing closures (of lifetime 'a) will require a mutable reference to the VM,
  • calling closures will require a reference (mutable when calling stored FnMut's, or or immutable when calling stored Fn's) to the VM.

I think that invariance is needed when storing the closures can be done with an immutable reference to the VM (just like PhantomData<Cell<T>> should be invariant over T, I assume).

However, all of this is still very confusing to me, and I will need a lot of time yet to better understand what happens, especially with the borrow checker. Maybe going over simpler examples over and over again might do the trick for me :sweat_smile:. (But also happy if someone has a good link to a tutorial to better understand borrowing, variance, covariance, reborrowing, etc.)

I've only lightly skimmed (the last few posts of) the discussion in this thread, but the examples here don't involve any scope-style functions that only introduce a handle by-reference in a call-back (in order to guarantee that all handles are dropped (before their lifetime parameter ends)). The unsoundness of a covariant handle on the other thread was based on the same potential soundness issue that prevented you from having an ordinary constructor in the first place (instead of that scope function expecting a callback).

TL;DR, I see an ordinary constructor here, no function with callbacks, so the situation is significantly different.

1 Like

I assume the fact that the example worked at all is also due to non-lexical lifetimes.

Thanks!

I'm currently tempted to make any calls that run code in the VM work on &mut machine. That way, I can also (safely) invoke FnMut's from the scripting language. Under these assumptions, I believe the Machine can be covariant over the lifetime parameter (just like a Vec<T> is covariant over T because you can only modify it when having a mutable reference). But I'm still experimenting.


Update: Apparently making such a VM operate on &mut self causes some trouble. Consider a simple case such as:

struct Machine {
    s: String,
}

impl Machine {
    fn getstr(&mut self) -> &str {
        &self.s
    }
}

fn main() {
    let mut m = Machine {
        s: "Hello".to_string(),
    };
    let _s1 = m.getstr();
    let _s2 = m.getstr();
    // Uncommenting the following line is an error:
    //println!("{} {}", _s1, _s2);
}

(Playground)

That's one of the setbacks I was talking about here :sweat_smile:. I had to write down this tiny example to be sure how things really are.

I can fix that by using a lot of Rc and Weak, etc., but possibly using interior mutability (i.e. making all operations on the VM require only an immutable/shared reference) makes things easier, because I can return values with a reasonable lifetime (that lasts while the machine is existent). And then I need invariance.

But I'm still unsure which variant is best. The whole task seems to be pretty difficult to judge about, maybe because I'm attempting to connect a very strict language with lifetimes (Rust) with a language that is highly dynamic and uses garbage collection (Lua). It feels like a real mismatch, but I'm determined to find a solution that works nicely.

I followed some of the advice here:

So I went ahead, but I ran into some annoying issues when demanding invariance. Let me provide a boiled-down example:

use std::marker::PhantomData;

#[derive(Default)]
struct Machine<'a> {
    _phantom: PhantomData<fn(&'a ()) -> &'a ()>,
}

struct Value<'a> {
    _machine: &'a Machine<'a>,
}

impl<'a> Machine<'a> {
    fn get_value(&'a self) -> Value<'a> {
        Value { _machine: self }
    }
}

// I cannot uncomment this:
/*
impl<'a> Drop for Machine<'a> {
    fn drop(&mut self) {}
}
*/

fn main() {
    let m: Machine = Default::default();
    let v = m.get_value();
    drop(v);
    // And I cannot uncomment this:
    //drop(m);
}

(Playground)

I think this is related to having the same lifetime ('a) in the reference and lifetime argument in _machine: &'a Machine<'a>, and the fact that the borrow of the Machine has a slighly shorter lifetime than the 'a of the lifetime argument.

I can fix this by using two lifetimes:

-struct Value<'a> {
+struct Value<'a, 'b> {
-    _machine: &'a Machine<'a>,
+    _machine: &'b Machine<'a>,
 }
 
 impl<'a> Machine<'a> {
-    fn get_value(&'a self) -> Value<'a> {
+    fn get_value<'b>(&'b self) -> Value<'a, 'b> {
         Value { _machine: self }
     }
 }

But this feels a bit annoying because the lifetimes basically will be almost the same in all cases.

I could also fix this by using my favorite technique of lifetime transmutation :innocent: :innocent: :innocent: (and shadowing the owned Machine):

 fn main() {
     let m: Machine = Default::default();
+    let m = unsafe { std::mem::transmute::<&Machine, &Machine>(&m) };

But I don't want to do that in productive code, of course.

So is there any clean(er) way to work around my problem of introducing an extra lifetime 'b that is basically (almost) the same as 'a? It would mean changing dozens of data types to accept an extra lifetime.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.