Using async closures with mut ref

help
1

I'm trying to understand why the following code doesn't work:

#![feature(async_closure)]

use core::future::Future;

async fn something() {}

#[derive(Default)]
struct Test {}

struct Int {
    i: i32,
}

impl Test {
    async fn perform<F>(&self, action: impl FnOnce(&mut Int) -> F) where F: Future<Output = ()> {
        let mut i = Int { i: 0 };
        let i_ref: &mut Int = &mut i;
        action(i_ref).await;
    }
}

fn main() {
    tokio::runtime::Runtime::new().unwrap().block_on(async {
        let test: Test = Default::default();
        let mut other = Int { i: 0 };
        let other_ref: &mut Int = &mut other;
        test.perform(async move |i: &mut Int| {
            something().await;
            
            // Why is this okay?
            other_ref.i = 1;
            
            // But not this? Commenting out fixes the code...
            i.i = 1;
        }).await;
    });
}

I get this error:

error: lifetime may not live long enough
  --> src/main.rs:27:22
   |
27 |         test.perform(async move |i: &mut Int| {
   |                      ^^^^^^^^^^^^^^^-^^^^^^^-
   |                      |              |       |
   |                      |              |       return type of closure is impl std::future::Future
   |                      |              let's call the lifetime of this reference `'1`
   |                      returning this value requires that `'1` must outlive `'2`

error: aborting due to previous error

error: could not compile `playground`.

The thing that confuses me is why the compiler is happy with other_ref that gets captured, but not with the mut ref being passed as parameter. Once I've run action(i_ref).await, I don't need i or i_ref anymore. Similar to how once I've run test.perform(...).await, I don't need other or other_ref.

2

Identifying the problem

async move |i: &'_ mut Int| {
    ...
}

is sugar for

//                 +---------+
//                 |         | captured inside the `async { ... }` Future.
//       vvvvvvvvvvv         v
move |i: &'_ mut Int| async move {
    ...
}

That is, is we call 'i the lifetime of the borrow over i, the returned async move { ... } block / impl Future type is infected with that 'i lifetime.

  • Returned type is of the form: CompilerGeneratedFutureFromAsyncBlock<'i, ...>

And that is not what the impl FnOnce(&mut Int) -> F where F : Future... expresses.

If we forget about the F : Future bound, the trait bound imposed on the closure is:

(&'_ mut Int) -> F
// i.e.
// there exists some single type `F` so that
for /* all */ <'i> (&'i mut Int) -> F

Now, a type F infected with a <'i> lifetime (such as in your actual call) will never be able to meet that criteria:

  • if F = CompilerGeneratedFutureFromAsyncBlock<'i, ...>, then when 'i changes, so does F, so the signature of the closure does not match that of a fixed return type _w.r.t. 'i.

Another way of seeing this, is that the (&'_ mut Int) -> F signature is expressing that F does not depend on the borrow of the input Int, which means that the following would type check:

fn does_typecheck<F> (
    closure: impl FnOnce(&mut Int) -> F,
) -> F
where
    F : Future,
{
    {
        let mut int: Int = ...;
        let f = closure(&mut int);
        f
    } // drops(int)
} // returns f

And then, if your actual closure happened to be accepted for this signature, the returned future would be dereferencing a dangling &mut Int when .await-ed the first time.

Fixing the problem

Now that we have identified the issue (F does not depend on the 'i lifetime of the input &'i mut Int), we can fix it.

  • fn perform<'i, F>... would allow F to depend on 'i, but at the cost of requiring an outer lifetime parameter, which by design must outlive the return point of the function, which your let mut i short-lived Int does not: you were correct in requiring what is called a HRTB (higher-rank trait bound): impl for<'i> FnOnce(&'i mut Int) ....

  • In an ideal world, the solution would be being able to write something along the lines of:

    impl for<'i> FnOnce(&'i mut Int) -> F<'i>,
where
    for<'i> F<'i> : Future<Output = ()>,

  • But that doesn't work with a generic parameter F, since F is expected to be a type itself / Rust does not accept feeding <'param> to type parameters (that would require HKT (higher-kinded types), or, if we allow ourselves to drop some sugar and use intermediate types, a weaker form of it, GAT, (generic associated types) should theoretically solve this. In practice, however, it doesn't: the trait solver isn't currently smart enough to reason at that level of abstraction, and it gets confused.

This means that in practice, we can no longer be that generic (over F I mean). We'll have to use a concrete type. But what kind of concrete type can be used with an async block? Well, the dynamic / runtime-dispatched type-erased future: Pin<Box<dyn Future<Output = ()> + 'lifetime>>, conventienty aliased as BoxFuture<'lifetime, ()> within the ::futures crate:

impl Test {
    async fn perform (
        self: &'_ Self,
        action: impl for<'i> FnOnce(&'i mut Int) -> BoxFuture<'i, ()>,
    )
    {
        let mut i = Int { i: 0 };
        let i_ref: &mut Int = &mut i;
        action(i_ref).await;
    }
}

#[::tokio::main]
async fn main ()
{
    let test: Test = Default::default();
    let mut other = Int { i: 0 };
    let other_ref: &mut Int = &mut other;
    test.perform(move |i: &mut Int| Box::pin(async move {
        something().await;
        
        // Why is this okay?
        other_ref.i = 1;
        
        // But not this? Commenting out fixes the code...
        i.i = 1;
    })).await;
}

Sadly the above now fails with:

error[E0597]: `other` does not live long enough
  --> src/main.rs:29:31
   |
29 |       let other_ref: &mut Int = &mut other;
   |                                 ^^^^^^^^^^ borrowed value does not live long enough
30 |       test.perform(move |i: &mut Int| Box::pin(async move {
   |  _____________________________________-
31 | |         something().await;
32 | |         
33 | |         // Why is this okay?
...  |
37 | |         i.i = 1;
38 | |     })).await;
   | |______- returning this value requires that `other` is borrowed for `'static`
39 |   }
   |   - `other` dropped here while still borrowed

Why? Well, while we have fixed the issue with the 'i lifetime, your closure also happens to capturing an outer ref that is not 'static, so the closure itself is also infected with an 'outer lifetime, which, in turn, infects the future too.

And the problem is, given the HRTB signature we have used, when 'i = 'static (even though it shall never be the case in practice), the future must be allowed to be + 'static, meaning that if it captures any 'outer thing, then 'outer >= 'static, so 'outer = 'static. So, again, we have overconstrained our signature, and it does not accept your use case.

Fixing this, in theory, could be easy:

  • we start by adding a classic lifetime parameter <'outer> to our perform function, and require that both the closure, and the future it generates outlive it:

    • the closure: impl 'outer + FnOnce...

    • the future?

  • BoxFuture only takes a single lifetime parameter, and 'i + 'outer isn't a lifetime parameter.

    • If we replace the type alias by its associated PinBox-ed trait object we should be able to solve this, right?

      Pin<Box<dyn 'i + 'outer + Future<Output = ()>>> // must outlive _both_ `'i` and `'outer`

      Well, it turns out, that for some reason / arbitrary limitation, trait objects cannot carry multiple lifetime bounds, even though it would be the perfect solution here :weary:

    • Adding a where 'outer : 'i (meaning 'outer >= 'i) and then using 'outer in our BoxFuture would solve everything, but we cannot have where bounds with for<...> parameters yet ... :weary:

At this point, I've been quite bummed that for such a simple initial problem we have reached some form of language limitation deadlock / dead end...

Except ... that implicit where bounds can be applied to all parameters, including for ones!

This means that the following works :sweat_smile:

/// Introduce the implicit 'snd : 'fst bound
struct Hack<'short, 'long : 'short, R = ()> (
    BoxFuture<'short, R>,
    PhantomData<&'long  ()>,
);

impl Test {
    async fn perform<'other>(
        self: &'_ Self,                                   // where 'other : 'i 🤯
        action: impl 'other + for<'i> FnOnce(&'i mut Int) -> Hack<'i, 'other, ()>,
    )
    {
        let mut i = Int { i: 0 };
        let i_ref: &mut Int = &mut i;
        action(i_ref).0.await;
    }
}

#[::tokio::main]
async fn main ()
{
    let test: Test = Default::default();
    let mut other = Int { i: 0 };
    let other_ref: &mut Int = &mut other;
    test.perform(move |i: &mut Int| Hack(Box::pin(async move {
        something().await;
        
        other_ref.i = 1;
        
        i.i = 1;
    }), PhantomData)).await;
}
6 Likes
3

@Yandros. Thanks for that answer. Really helpful to see it spelled out like that.

Another way around the issue based on the following comment

Just change the closure argument to Box<Int>. If you just want to pass Int by a reference, Box is easier than & since it passes ownership on the call. & requires that the borrow lifetimes make sure that you can access the original location in memory (which is what @Yandros somehow managed to work around :slight_smile: ) . Box also is just a pointer to a value, but it has the benefit of that knowing that its the only pointer to that value, so when its dropped, it knows it can safely deallocate the memory it points to, which simplifies a lot of reasoning as far as the compiler is concerned.


impl Test {
    async fn perform<F>(&self, action: impl FnOnce(Box<Int>) -> F) where F: Future<Output = ()> {
        let i = Int { i: 0 };
        let i_ref = Box::new(i);
        action(i_ref).await;
    }
}

fn main() {
    tokio::runtime::Runtime::new().unwrap().block_on(async {
        let test: Test = Default::default();
        let mut other = Int { i: 0 };
        let other_ref: &mut Int = &mut other;
        test.perform(async move |mut i: Box<Int>| {
            something().await;
            
            // Why is this okay?
            other_ref.i = 1;
            
            // But not this? Commenting out fixes the code...
            i.i = 1;
        }).await;
    });
}

Playground

There are some other differences between the two solutions in terms of memory placement so if its performance critical you can try both approaches. The reference approach in your code I believe keeps the reference to a location on the stack of perform (though that might not be true with tokio) whereas Box puts it on the heap. Though Box isn't guaranteed to avoid first initializing the data on the stack and copying to the heap, but there are ways around that such as the copyless crate if needed.

2 Likes
4

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.