Ephemeral (or "weak") closures

Continuing the discussion from Passing Rust closures to C:

I created a module which allows converting closures to closures with a longer lifetime. I originally needed this to create a nicer interface when passing Rust closures to Lua (as I didn't like the requirement to make all Rust closures having a 'static lifetime or the lifetime of the Lua virtual machine.

The trick is to have a scope handle, similar to what crossbeam or rayon do with their scoped threads.

Throughout the discussion in the above linked thread, I came to the conclusion that the mechanism may be of broader interest and not just be limited to (Lua) sandboxing. That's why I came up with an own module (that I would like to turn into a crate eventually).

For the context:

This is what I came up with, so far:

use std::marker::PhantomData;
use std::mem::transmute;
use std::sync::Arc;

pub struct Scope<'scope> {
    arc: Arc<()>,
    phantom: PhantomData<fn(&'scope ()) -> &'scope ()>,
}

pub fn scope<'scope, O, R>(run: O) -> R
where
    O: FnOnce(&Scope<'scope>) -> R,
{
    run(&Scope::<'scope> {
        arc: Default::default(),
        phantom: PhantomData,
    })
}

impl<'scope> Scope<'scope> {
    pub fn extend_mut<'long, A, R, E, F>(
        &self,
        mut error: E,
        mut func: F,
    ) -> Box<dyn 'long + FnMut(A) -> R>
    where
        'long: 'scope,
        A: 'long,
        R: 'long,
        E: 'long + FnMut(A) -> R,
        F: 'scope + FnMut(A) -> R,
    {
        let weak = Arc::downgrade(&self.arc);
        let func_or_error = Box::new(move |arg: A| -> R {
            match weak.upgrade() {
                Some(_) => func(arg),
                None => error(arg),
            }
        });
        unsafe {
            transmute::<
                Box<dyn 'scope + FnMut(A) -> R>,
                Box<dyn 'long + FnMut(A) -> R>,
            >(func_or_error)
        }
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use std::sync::Mutex;
    #[test]
    fn test_in_scope() {
        let errorstate: Arc<Mutex<i32>> = Arc::new(Mutex::new(0));
        let clone = errorstate.clone();
        let error = move |_| {
            *clone.lock().unwrap() += 3;
        };
        let mut i: Box<i32> = Box::new(0);
        scope(|s| {
            let mut closure = s.extend_mut(error, |_| {
                *i += 2;
            });
            closure(());
        });
        assert_eq!(*i, 2);
        assert_eq!(*errorstate.lock().unwrap(), 0);
    }
    #[test]
    fn test_out_of_scope() {
        let errorstate: Arc<Mutex<i32>> = Arc::new(Mutex::new(0));
        let clone = errorstate.clone();
        let error = move |_| {
            *clone.lock().unwrap() += 7;
        };
        let mut i: Box<i32> = Box::new(0);
        let mut closure = scope(|s| {
            let closure = s.extend_mut(error, |_| {
                *i += 5;
            });
            closure
        });
        closure(());
        drop(closure);
        assert_eq!(*i, 0);
        assert_eq!(*errorstate.lock().unwrap(), 7);
    }
    #[test]
    fn test_out_of_scope_with_drop() {
        let errorstate: Arc<Mutex<i32>> = Arc::new(Mutex::new(0));
        let clone = errorstate.clone();
        let error = move |()| {
            *clone.lock().unwrap() += 13;
        };
        let mut i: Box<i32> = Box::new(0);
        let mut closure = scope(|s| {
            let closure = s.extend_mut(error, |_| {
                *i += 11;
            });
            closure
        });
        assert_eq!(*i, 0);
        drop(i);
        closure(());
        assert_eq!(*errorstate.lock().unwrap(), 13);
    }
}

(Playground)

Output:


running 3 tests
test tests::test_out_of_scope ... ok
test tests::test_in_scope ... ok
test tests::test_out_of_scope_with_drop ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


A workaround could be to pass a single tuple and to provide another closure which unwraps the tuples. Or the crate could provide methods like extend_mut_1arg_1retval, extend_mut_1arg_2retvals extend_mut_1arg, extend_mut_2args, etc. Neither option seems nice though.

I kept that consideration in mind and managed to use only one indirection. See Playground example above, where the closure that performs the match on the Arc works directly with the provided closures, and the lifetime transmutation happens afterwards. Thanks for your hint!

Also many thanks to @Yandros for pointing out the problem of covariance/invariance regarding scope handles:

I hope I adequately handled this problem by including a PhantomData<fn(&'scope ()) -> &'scope ()> in my scope handle.

I would be happy about a short review and comments on whether the unsafe in my above code is sound.

1 Like

My initial intuition before even looking at your code: The answer is probably "no" :⁠-P -- there's just soo many ways in which such an API could be unsound.

My answer after looking at the code: One thing you missed is destructors! Your soundness seems to rely on not calling func outside of its lifetime; but you're still dropping it outside of its lifetime! Let me quickly try to come up with a concrete exploit/demonstration....

Here we go:

fn main() {
    let g = {
        let string = "Hello World".to_owned();
        scope(|s| {
            struct PrintOnDrop<'a>(&'a str);
            impl Drop for PrintOnDrop<'_> {
                fn drop(&mut self) {
                    println!("{}", self.0);
                }
            }
            let r = PrintOnDrop(&string);
            let f = move |()| {
                &r;
            };
            s.extend_mut(|()| (), f)
        })
    };
    drop(g); // prints contents of `string` (use-after-free)
}

(playground)

I'm not sure if there's any (good) way to save this besides something like requiring F: Copy.

5 Likes

I have an idea. How about storing the original closure inside the Arc? I'm not happy with the following solution yet, due to the Mutex being locked while func is run, but at least this should not cause any undefined behavior (I hope):

use std::marker::PhantomData;
use std::mem::transmute;
use std::sync::{Arc, Mutex};

pub struct Scope<'scope> {
    closures: Arc<Mutex<Vec<Box<dyn 'scope + FnMut()>>>>,
    phantom: PhantomData<fn(&'scope ()) -> &'scope ()>,
}

pub fn scope<'scope, O, R>(run: O) -> R
where
    O: FnOnce(&Scope<'scope>) -> R,
{
    run(&Scope::<'scope> {
        closures: Default::default(),
        phantom: PhantomData,
    })
}

impl<'scope> Scope<'scope> {
    pub fn extend_mut<'long, E, F>(
        &self,
        mut error: E,
        func: F,
    ) -> Box<dyn 'long + FnMut()>
    where
        'long: 'scope,
        E: 'long + FnMut(),
        F: 'scope + FnMut(),
    {
        let weak = Arc::downgrade(&self.closures);
        let func = Box::new(func);
        let mut guard = self.closures.lock().unwrap();
        let index = guard.len();
        guard.push(func);
        let result: Box<dyn 'scope + FnMut()> =
            Box::new(move || match weak.upgrade() {
                Some(closures) => {
                    let mut guard = closures.lock().unwrap();
                    let func = &mut guard[index];
                    func()
                }
                None => error(),
            });
        let result: Box<dyn 'long + FnMut()> =
            unsafe { transmute(result) };
        result
    }
}

fn main() {
    let g = {
        let string = "Hello World".to_owned();
        let g = scope(|s| {
            struct PrintOnDrop<'a>(&'a str);
            impl Drop for PrintOnDrop<'_> {
                fn drop(&mut self) {
                    println!("{}", self.0);
                }
            }
            let r = PrintOnDrop(&string);
            let f = move || {
                &r;
            };
            let g = s.extend_mut(|| (), f);
            println!("About to leave scope");
            g
        });
        println!("Scope has been left");
        g
    };
    println!("Dropping `g` is safe now");
    drop(g);
}

(Playground)

Output:

About to leave scope
Hello World
Scope has been left
Dropping `g` is safe now


Amendment:

Note that I switched the signature for the closure to be converted from FnMut(A) -> R to FnMut() to keep things simpler.

Maybe the Mutex could be released earlier with something like that:

         let result: Box<dyn 'scope + FnMut()> = Box::new(move || match weak.upgrade() {
             Some(closures) => {
                 let mut guard = closures.lock().unwrap();
-                let func = &mut guard[index];
-                func()
+                let func: *mut Box<dyn 'scope + FnMut()> = &mut guard[index];
+                drop(guard);
+                unsafe { (*func)() }
             }
             None => error(),
         });

Amendment #2:

Here everything put together in a new Playground including a test function that checks whether the dropping happens once the scope handle is dropped.

I'm not entirely sure why you're using Arc and Mutex when the returned Box<dyn FnMut…> is not Send anyways. Also note that the absense of Send here is crucial for soundness, otherwise you could be in the middle of a function call while the scope is being left, resulting in the unsound drop again.

2 Likes

One possible catch I might have missed is the drop order of the Vec (see also Drop Check). But I think it's sound: Method extend_mut expects a closure of type F: 'scope + FnMut(). I cannot pass a closure with a shorter lifetime 'a, thus any generic arguments of captured types must strictly outlive 'scope (I think).

If I understand it right, it's not harmful to use Arc instead of Rc, but just unnecessary overhead (assuming my returned closure isn't Send anyways).

Maybe this could this be fixed with something like:

impl<'scope> Drop for Scope<'scope> {
    fn drop(&mut self) {
        drop(self.closures.lock().unwrap());
    }
}

However, then I must hold some sort of lock (maybe a ReadGuard) during the execution of the closure, which I just circumvented earlier with the drop(guard); unsafe { (*func)() }.

I'll try to come up with a new example eventually, where I either use Rc instead of Arc or use some sort of locking to make sure the destructors aren't run until all closures have finished their execution.

yes

I put it together, and this is the result (comments added):

(Playground)

Again, I'd be happy about a hint where the code may be broken (or encouragement regarding my unsure assessment that it's sound :innocent:).

However, I have to admit that I started this thread believing I could come up with a simple solution that's easy to check. Things have gotten more complex since then. The current solution

  • uses two locks now (a Mutex and an RwLock),
  • defers dropping by acquiring a lock,
  • doesn't support arguments anymore,
  • feels very "twisted" overall.

I still believe it would be nice to have such a crate that allows extending a closure's lifetime. I don't feel like my implementation is very elegant (and not even sure if it's sound, which is why I'd appreciate your feedback).

I feel like I might defer this issue regarding my own application case (providing closures to a sandbox that executes interpreted languages). Nonetheless, I find the idea of ephemeral/weak closures very interesting and would like to experiment further on it to get to a working (and safe) solution. Or to encourage others to create such a crate :grin:.

As a sanity check, the type of extended should have the + Send as-well. It’s enough if the transmute changes the lifetime.

I’m currently looking for remaining obvious soundness issues. I have a feeling the current mechanism using droplock.write() when dropping InnerScope might be useless since a currently-running closure would hold onto a strong copy of the Arc<InnerScope<_>> itself, preventing InnerScope to be dropped when exiting the scope in the first place.

1 Like

Yep, something like

use std::sync::mpsc;
use std::thread;
use std::time::Duration;

fn main() {
    let string = "Hello World".to_owned();
    let (tx1, rx1) = mpsc::channel::<()>();
    let (tx2, rx2) = mpsc::channel::<()>();
    let handle = scope(|s| {
        let string_ref = string.as_str();
        let mut f = s.extend_mut(
            || (),
            move || {
                tx1.send(()).unwrap();
                // use timeout to avoid deadlock in case `scope`
                // was correctly implemented
                let _ = rx2.recv_timeout(Duration::from_secs(1)); // ignore timeout
                println!("{}", string_ref);
            },
        );
        let handle = thread::spawn(move || {
            f();
        });
        rx1.recv().unwrap();
        handle
    });
    drop(string);
    let _ = tx2.send(()); // ignore disconnected
    handle.join().unwrap();
}

uses the string after free.

2 Likes

Thank you very much! :+1:

Phew. :hot_face:

I could try once more to fix that, but I feel like I might easily make more mistakes here. The overall task seems hard to solve – maybe even impossible, as you suggested here:

But even if it could be solved, there are still some other problems:

  • being generic over the functions arguments (including arity) and return values,
  • supporting Send and !Send (which, I guess, requires two different extend_mut methods?),
  • supporting Fn and FnOnce in addition to FnMut with methods like extend, extend_once, extend_send, extend_once_send, etc. (resulting in 6 different methods for each arity when supporting Send and !Send, I think?),
  • a lot of Boxing (of which some is maybe unnecessary?),
  • overall complexity of proving that a solution where nobody finds an error really is free of errors (certainly something I could not do with my current knowledge and experience).

Giving up the idea of making a generic crate (and instead just using Arcs in each concrete case to make my closures 'static, for example) might be the path with least effort for me, for now.

Anyway, if someone has a nice (generic as possible) solution for the problem, I'm interested, of course.

Thanks again to everyone giving me hints and advice, and this review, of course.


P.S.: Maybe the previous version that is !Send works, though (and could be simplified to use Rc instead of Arc). But even if it works, it would always be !Send, and most of the problems listed above would apply as well.

I couldn't rest and had to think about this again :weary:

This led me to consider trying something more easy:

  • store all closures directly in the scope handle
  • move a (wide) pointer to the extended closure to allow calling the original closure from there (while keeping the dropping managed entirely by the scope handle)
  • Use an Arc<RwLock<bool>> to syncronize execution of closure and dropping of scope

Here is my code (Playground) including comments and my original tests.

With my new approach, the string will be printed before it's dropped: (Playground)

Output:

Hello World
DROPPED


Amendment: It's possible to be generic over a (single) argument and return value by using an extra indirection. See modified example here (Playground), where the closures can retrieve an argument and return a value (requires double-boxing in line 57 though).

(Assuming the whole thing is safe now… :sweat_smile:)

1 Like

Now, I've tried to abstract the concept of something like an Arc<Foo<'a>> where you can hold a Weak reference into it without needing to mention the lifetime; and with a scope that makes sure that the the contents are dropped before the scope is left (so that only currently unused weak pointers remain). For now, I came up with something like

////////////////////////////////////////////////////////////////////////////////
// This section from https://docs.rs/existential/0.1.1/src/existential/lib.rs.html#106-113

pub trait TyConFor<'a> {
    type Applied;
}

pub type Apply<'a, C> = <C as TyConFor<'a>>::Applied;

pub trait TyCon: for<'a> TyConFor<'a> {}
impl<C: ?Sized> TyCon for C where C: for<'a> TyConFor<'a> {}

////////////////////////////////////////////////////////////////////////////////

pub mod scoped_existential {
    use super::*;

    use std::mem;
    use std::sync::{Arc, RwLock, TryLockError, Weak};

    pub fn scope<'scope, F, R>(f: F) -> R
    where
        F: FnOnce(&mut Scope<'scope>) -> R,
    {
        f(&mut Scope {
            data: <_>::default(),
            lock: <_>::default(),
        })
    }

    pub struct Scope<'scope> {
        data: Vec<Arc<dyn Send + Sync + 'scope>>,
        lock: Arc<RwLock<()>>,
    }

    impl<C: TyCon> Clone for ScopedExistential<C> {
        fn clone(&self) -> Self {
            Self {
                payload: self.payload.clone(),
                lock: self.lock.clone(),
            }
        }
    }

    // TODO: Send/Sync implementation is slightly inaccurate (due to 'static)
    pub struct ScopedExistential<C: TyCon> {
        payload: Weak<Apply<'static, C>>,
        lock: Arc<RwLock<()>>,
    }

    impl<C: TyCon> ScopedExistential<C> {
        pub fn with_upgrade<F, R>(&self, f: F) -> R
        where
            F: FnOnce(Option<&Apply<'_, C>>) -> R,
        {
            match self.lock.try_read() {
                Err(TryLockError::Poisoned(_)) => unreachable!(),
                Err(TryLockError::WouldBlock) => f(None),
                Ok(guard) => match self.payload.upgrade() {
                    None => {
                        drop(guard);
                        f(None)
                    }
                    Some(payload) => {
                        let r = f(Some(&payload));
                        drop(payload);
                        drop(guard);
                        r
                    }
                },
            }
        }
    }

    impl<'scope> Scope<'scope> {
        /// erases lifetime "'a" from a type Arc<Foo<'a>>
        pub fn erase<'a: 'scope, C: TyCon>(&mut self, val: Apply<'a, C>) -> ScopedExistential<C>
        where
            C: 'scope,
            Apply<'a, C>: Send + Sync,
        {
            let val = Arc::new(val);
            let weak = Arc::downgrade(&val);
            self.data.push(val);
            let payload =
                unsafe { mem::transmute::<Weak<Apply<'a, C>>, Weak<Apply<'static, C>>>(weak) };
            ScopedExistential {
                payload,
                lock: self.lock.clone(),
            }
        }
    }

    impl<'scope> Drop for Scope<'scope> {
        fn drop(&mut self) {
            let guard = self.lock.write().unwrap();
            self.data.clear();
            drop(guard);
        }
    }
}

and this seems to allow re-implementing your API without further unsafe code: Rust Playground

2 Likes

Very nice!! :smiley:

I don't understand most your code yet (particularly difficult for me with the code from the existential crate, but I'll try to understand in time). The code passes my tests and your fn main, really nice!

If it works, it would also allow me to use it for a concrete interface for my Lua machine (where I might not even need Send or being generic about the closures argument(s) and return value). And unsafe code would be in a seperate module or crate (hopefully published by someone else, so I'm not to blame :innocent:).

(Btw, I just learned you can have tests and a main function on Rust Playground.)

Without having fully understood your code yet, I noticed that you omitted the invariant PhantomData for 'scope. Is that intended? I remember I had some problems when I left it out, but to be honest, I get confused over this. I included it due to having problems and @Yandros' hint in this post. Unfortunately I don't remember the problem I ran into (Edit: if there was one at all).


See also crossbeam::Scope, where a PhantomData with invariance is used. I still try to remember if I had any problem when leaving it out and if there was one at all, but can't reproduce any error if I take it out from my code.

Semi-intended. I initially thought about completely preventing swaps and other shenanigans by only providing a Pin<&mut Self> for the Scop<'scope>, but then I came to the conclusion that not disallowing those might actually not be unsound. The variance was irrelevant with the Pin-involving design; I forgot to think about it again.. then later (actually after already posting my previous answer) considering the question again, I notived the difference in variance compared to your Scope, but couldn't find any way in which the lifetime being covariant is unsound.

Edit: Not Now that I'm thinking about it again, there probably is a soundness issue after all, if it's not invariant... I can try to give more details later.

Not sure if it makes a difference, but I just noticed that I changed the reference to the scope handle passed to the FnOnce closure (run in my case, f in your case, I think) from being immutable to being mutable. In @Yandros post as well as in case of crossbeam, an immutable reference is used.


Rethinking about this (it's really tough for me to understand it), if Scope<'scope> is covariant in 'scope, then (i think) 'scope is allowed to be shorter than the runtime of run (or f in your case). Thus, the bound F: 'scope + … (or C: 'scope in your erase method) is too lax. But not sure how to exploit it. But it does feel like there is a PhantomData with invariance needed to make it sound.

@jbe, so the problem when it's an immutable reference is simply that your safety condition breaks down since 'scope can become arbitrarily small, even though it's used as a lower bound on how long the closures must be able to live.

With a mutable reference, that's not all that easy anymore because mutable references themselves are immutable in the target type. And turning the mutable reference into an immutable one wouldn't help because you can't do anything with a &Scope<'scope> when all of Scope's methods require &mut self.

However, with API such as replace_with::replace_with_or_abort, you can take the Scope<'large_scope> out of a &mut Scope<'large_scope>, then convert that into a Scope<'small_scope>, and use it with mem::replace to avoid a nested/inner scope from being dropped. You won't be able to actually finish the outser (large_scope) scope anymore because you don't have a Scope<'large_scope> left anymore; but before the inevitable abort, you can still cause unsoundness by accessing stuff related to the inner scoped that wasn't dropped properly.

Now, even when it's invariant, you can still swap / replace stuff like this in weird ways, but only between scopes with identical lifetime argument; and I believe that whatever you do, there's still the invariant that: every Scope<'a> will be dropped before the lifetime 'a becomes invalid.

Just realized, you don't even need the second/inner scope(...) call. Just:

use std::mem;
fn main() {
    scope(|s| {
        replace_with::replace_with_or_abort(s, |s| {
            let string = "Hello World".to_owned();
            let string_ref = &string;
            let mut s = s; // variance!
            let mut f = s.extend_mut(|()| (), |()| {
                println!("{}", string);
            });
            mem::forget(s);
            drop(string);
            f(()); // accesses the string after free
            panic!(); // results in abort
        })
    });
}

(gist (not a playground because of replace_with dependency))

So to say it in very simple terms (give me time to understand all the details later :hot_face:), it's always wrong to omit the invariant PhantomData. In case of methods with an immutable reference to the scope handle it's just easier to construct / run into an example that is not sound?

(That might explain why I wasn't capable of reproducing the problems I experienced when removing the PhantomData again.)

P.S.: Or does that mean that replace_with::replace_with_or_abort would need to be marked unsafe?

P.P.S.: Or would the scope need to be marked !UnwindSafe somehow?

As I tried to explain above, it's not wrong if you're only providing Pin<&mut Scope<'scope> and Scope: !Unpin (which is something I did for some time). But in this case making it invariant wouldn't make a difference either. So it's never wrong to have the marker for invariance, and in most cases it's also necessary.

No, that API is usually considered safe as far as I know.

!UnwindSafe is merely a "lint" if you will; it can always be overridden by AssertUnwindSafe without unsafe code. Also unwind safety is about catching panics; the code example above doesn't catch any panic.