Working with Arc and FnMut

1

I'm writing a library that needs to move something that isn't copy into a FnMut. The problem is that this isn't allowed, as the function might be called twice and you can only move once. The thing in question is an Arc, so it's cloneable. Ideally I would like a struct like

struct CallbackAndArc<T> {
    value: Arc<T>,
    callback: Box<dyn Fn()>,
}

Where the fn has access to value. I know it's safe because when the value gets deallocated, the callback is also dealloc'd so cannot be called. This kinda looks like self-referential structs.

Is there any way of doing what I'm trying to do?

2

I apologize for the vagueness of the question, but I don't really know what I'm doing with this.

3

You could wrap the non-clone thing in a Arc, then clone the Arc and move that into the FnMut.

struct NotClone(Box<dyn Debug>);

fn main() {
    let a = NotClone(Box::new(10));
    let a = Arc::new(a);
    
    let func = {
        let a_clone = a.clone();
        move || {
            println!("{:?}", a_clone);
        }
    };
}

instead of creating a new struct for this.

Also moving in to a closure only if you consume the value inside the closure, so the closure func above is Fn, because it doesn't change anything inside the body, even though a_clone is moved into the closure.

2 Likes
4

I think this is what I want!

5

The problem I was having in the end was that I was moving the arc inside the closure, which I guess meant that it was no longer around for the next call. The solution was to clone it.

I really feel like I do not understand closures. Normally when something goes out of scope at the end of a function it gets Dropped, but this doesn't seem to happen with closures.

6

note: I have turned this into a blog post and added some more information, you can find it here:

The first thing to understand about closures is that they are pure sugar, and three traits working together.

The three traits are
note I have removed some unnecessary details, like function calling convention

trait FnOnce<Args> { type Output; fn call_once(self, args: Args) -> Self::Output; }
trait FnMut<Args> : FnOnce<Args> { fn call_mut(&mut self, args: Args) -> Self::Output; }
trait Fn<Args> : FnMut<Args> { fn call(&self, args: Args) -> Self::Output; }

I will show the desugaring of a few closures. Note: I will not show how Send and Sync are impled, as that is out of scope. After the first desugaring, I will not show the impls for all three Fn* traits, only the most specific one. So if you see Fn, then assume FnMut and FnOnce are impled with the same function body. If you see FnMut, then assume that FnOnce is impled with the same function body, but Fn is not impled. If you see FnOnce, then assume that Fn and FnMut are not impled. I will also put type Output in a comment to show what it would be if I only impl Fn or FnMut.

First, the simplest closure, one that doesn't capture anything, and only returns a unit.

let x = || ();
let y = x();

gets desugarred to

#[derive(Clone, Copy)]
struct __closure_0__ {}

impl FnOnce<()> for __closure_0__ {
    type Output = ();
    
    fn call_once(self, args: ()) -> () {
        ()
    }
}

impl FnMut<()> for __closure_0__ {
    fn call_mut(&mut self, args: ()) -> () { () }
}

impl Fn<()> for __closure_0__ {
    fn call(&self, (): ()) -> () { () }
}

let x = __closure_0__ {};
let y = x.call(());

Rust derives Clone and Copy to the closure because it can. This is to allow flexibility when using a closure.

Rust will pick the most specific Fn* trait to use whenever you call a function, in this order: Fn, FnMut, FnOnce.

Note: the names I give, __closure_0__ are arbitrary and the names that are actually used are random. So closures are unnameable.

Note: How Rust knows which Fn* trait to derive for the closure is up to analysis of what it captures and how it is used (seen later).

Now one step up, lets capture a variable.

let message = "Hello World!".to_string();
let print_me = || println!("{}", message);

print_me();

desugars to

#[derive(Clone, Copy)]
struct __closure_1__<'a> { // note: lifetime parameter
    message: &'a String, // note: &String, not String
}

impl<'a> Fn<()> for __closure_1__<'a> {
    // type Output = ();
    fn call(&self, (): ()) -> () {
        println!("{}", self.message)
    }
}

let message = "Hello World!".to_string();
let print_me = __closure_1__ { message: &message };

print_me.call(());

Notice the lifetime parameter on __closure_1__, because it is borrowing from the stack frame with &message, print_me has a non-'static lifetime. So it can't be sent across threads!

Now, what about if I have a closure with arguments? What about move closures?

let header = "PrintMe: ".to_string();
let print_me = move |message| println!("{}{}", header, message);

print_me("Hello World!");

desugars to

#[derive(Clone)]
struct __closure_2__ { // note: no lifetime parameter
    header: String // note: String, not &String
}

impl<'a> Fn<(&'a str,)> for __closure_2__ {
    // type Output = ();
    fn call(&self, (message,): (&'a str,)) {
        println!("{}{}", self.header, message);
    }
}

let header = "PrintMe: ".to_string();
let print_me = __closure_2__ {
     header: header // note: no &
};

print_me.call(("Hello World!",));

Notice that the args are defined as tuples. This is a hack to get any number of args with a single type parameter, and this is how it is desugared right now. This desugaring is unstable and could change.

Notice that with move, we eliminated all references in __closure_2__. Because of this, __closure_2__ always has a 'static lifetime, which is necessary for it to be safely sent across threads! (There are more requirements such as Send and Sync, but that is out of scope). But in doing so, we also lost Copy, now our closure in only Clone.

This is why when you do anything with threads, you need to use move closures!

(This example is because of @cuviper's comment below)

More on move

let a = "Hello World".to_string();
let a_ref = &a;

let print_me = move || println!("{}", a_ref);

print_me();

desugars to

// lifetimes are back, even though this is a `move` closure
// because this closure captures a reference
// note: a new lifetime parameter will be created for
// user-defined structs that also have lifetime parameters.
#[derive(Clone, Copy)]
struct __closure_3__<'a> {
    a_ref: &'a String
}

impl<'a> Fn<()> for __closure_3__<'a> {
    // type Output = ();

    fn call(&self, (): ()) {
        println!("{}", self.a_ref)
    }
}

let a = "Hello World".to_string();
let a_ref = &a;

// because this is a move closure, there are no new references here
let print_me = __closure_3__ { a_ref: a_ref };

Notice that even though we have a move closure, we still get lifetimes because we have a reference. This means that unless that reference resolves to be 'static, you cannot send it across threads.

What about returning things from closures, and mutating the enviornment inside a closure.

let mut counter: u32 = 0;
let delta: u32 = 2;

let mut next = || {
    counter += delta;
    counter
};

assert_eq!(next(), 2);
assert_eq!(next(), 4);
assert_eq!(next(), 6);

desugars to

struct __closure_4__<'a, 'b> {
    counter: &'a mut u32,
    delta: &'b u32
}

impl<'a, 'b> FnMut<()> for __closure_4__<'a, 'b> {
    // type Output = u32;

    fn call_mut(&mut self, (): ()) -> u32 {
        *self.counter += *self.delta;
        *self.counter
    }
}

let mut counter: u32 = 0;
let delta: u32 = 2;

let mut next = __closure_4__ {
    counter: &mut counter,
    delta: &delta
};

assert_eq!(next.call_mut(()), 2);
assert_eq!(next.call_mut(()), 4);
assert_eq!(next.call_mut(()), 6);

What about consuming things in a closure

let a = vec![0, 1, 2, 3, 4, 5, 100];

// notice, no `move`
let transform = || {
    let a = a.into_iter().map(|x| x * 3 - 1);
    a.sum::<u32>()
};

println!("{}", transform());
// println!("{}", transform()); // error[E0382]: use of moved value: `transform`

desugars to

struct __closure_5__ {
    a: Vec<u32> 
}

impl FnOnce<()> for __closure_5__ {
    type Output = u32;
    
    fn call_once(self, (): ()) -> u32 {
        let a = self.a.into_iter().map(|x| x * 3 - 1);
        a.sum::<u32>()
    }
}

let a = vec![0, 1, 2, 3, 4, 5, 100];

let transform = __closure_5__ { a: a };

println!("{}", transform.call_once(()));
// println!("{}", transform.call_once(())); // error[E0382]: use of moved value: `transform`

With that, you should know everything about how a closure is desugared.

7 Likes
7

Nice summary!

In your example the closure does have a 'static lifetime, but that's not necessarily true of all move closures. It just changes the capturing mechanism itself to take the value instead of creating a reference. But if the value you're capturing is already a reference, or otherwise has some lifetime, then the closure will still be limited by that.

Also, the 'static lifetime is sort of orthogonal to thread safety. It's true that std::thread::spawn requires 'static, but rayon, crossbeam, and others provide APIs to have scoped threads. Send and Sync are the truly critical points for thread safety. For move and 'static, I would emphasize that this enables you to escape the immediate scope that would otherwise constrain the closure -- which could be used to send it to an unscoped thread, or even just return the closure from a function.

You need mut here, of course. :slightly_smiling_face:

1 Like
8

Thank You!

Yes I realize that, this post is still is progress, but thank you for pointing it out.

edit: I reworded that section so that it is more clear that 'static is necessary, but not sufficient.

9

FWIW, I wrote a reddit comment a while back also digging into closure desugaring. When you get your post "finished" to your satisfaction, it would be nice to see it published in a more permanent place, so we're not just dumping knowledge in random internet threads. :slight_smile:

2 Likes
10

Yes, that would be nice, I was thinking of making this a blog post.

2 Likes
11

I have converted the reply into a blog post!

1 Like
12

Brilliant, thanks so much for the write up!

I now realize my problem was that because I consumed something from the environment in my closure, the compiler only implemented FnOnce, where I needed FnMut. When I stopped consuming it, the compiler could generate the FnMut impl I was looking for.

I feel like I have a full understanding now of somthing I considered magical before. So thanks very much. I almost feel like the article should be in the official documentation, or at least rust-by-example or something.

1 Like