How to express that the Future returned by a closure lives only as long as its argument?

[KernelMC]

I'm trying to write an async function which takes a closure as an argument. The closure is to return a Future and to take a reference to a value which is local to the function to which it was passed. Here's my first attempt.

async fn access<T, Fut, F>(data: &RwLock<String>, accessor: F) -> T
where
    F: FnOnce(&str) -> Fut,
    Fut: Future<Output = T>,
{
    let guard = data.read().await;
    accessor(&guard).await
}

I thought this was fine until I tried to use it:

async fn test_access() {
    let data = RwLock::new("sehr problematisch".to_owned());
    let data_len = access(&data, |message| async move { message.len() }).await;
}

This has a lifetime error because the argument to the closure needs to live longer than the Future returned by it, but I didn't express this constraint in the signature of access.

error: lifetime may not live long enough
  --> src/lib.rs:15:44
   |
15 |     let data_len = access(&data, |message| async move { message.len() }).await;
   |                                   -------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ returning this value requires that `'1` must outlive `'2`
   |                                   |      |
   |                                   |      return type of closure `[async block@src/lib.rs:15:44: 15:72]` contains a lifetime `'2`
   |                                   has type `&'1 str`

error: could not compile `playground` due to previous error

(playground link: Rust Playground)

I thought I could fix this by adding lifetime parameters to access.

async fn access<'a, 'b, T, Fut, F>(data: &'b RwLock<String>, accessor: F) -> T
where
    'b: 'a,
    F: FnOnce(&'a str) -> Fut,
    Fut: 'a + Future<Output = T>,
{
    let guard = data.read().await;
    accessor(&guard).await
}

However this also failed with a lifetime error, this time in the body of access.

error[E0597]: `guard` does not live long enough
  --> src/lib.rs:11:14
   |
4  | async fn access<'a, 'b, T, Fut, F>(data: &'b RwLock<String>, accessor: F) -> T
   |                 -- lifetime `'a` defined here
...
11 |     accessor(&guard).await
   |     ---------^^^^^^-
   |     |        |
   |     |        borrowed value does not live long enough
   |     argument requires that `guard` is borrowed for `'a`
12 | }
   | - `guard` dropped here while still borrowed

For more information about this error, try `rustc --explain E0597`.

(playground link: Rust Playground)

This error makes sense. Because 'a has no constraints, the borrow checker assumes it lives forever, as the function could be called with 'static' substituted for this parameter.

What I'd like to do is express to the borrow checker that the value returned from the closure only lives as long as its argument, but without exposing a lifetime parameter to the callers of access. Are there any language features that allow me to express this?

(edit: fixed syntax highlighting)

[vague]

Solution for this post

And also see other posts:

• Asynchronous wrapper function taking a reference
• Lifetime bounds to use for Future that isn't supposed to outlive calling scope

[KernelMC]

In the context in which I'll be calling the real version of the access function, heap allocations are not acceptable. So I can't use BoxFuture as you suggested.

However, I did see in one of the links you posted that you created a trait which inherits from FnMut to express a lifetime constraint on a closure's argument and return value. This seems to be exactly what I need. I implemented it here, but I ran into a new compiler error.

error[E0644]: closure/generator type that references itself
  --> src/main.rs:23:34
   |
23 |     let data_len = access(&data, |message| async move { message.len() }).await;
   |                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cyclic type of infinite size
   |
   = note: closures cannot capture themselves or take themselves as argument;
           this error may be the result of a recent compiler bug-fix,
           see issue #46062 <https://github.com/rust-lang/rust/issues/46062>
           for more information

For more information about this error, try `rustc --explain E0644`.

I don't understand why the closure is capturing itself. I verified that access can be called with a top-level function as its second argument. Is there a way to get it to work with a closure as well?

[semicoleon]

I believe that's what's mentioned here

Lifetime bounds to use for Future that isn't supposed to outlive calling scope

I’ve written answers on similar problems in the past. E.g. here or here and the second one links some more places. It’s possible to avoid the need for Box using trait, and @vague above provided an example, too, but those will unfortunately (for no good reason) not work with closures, only with (async) fn items. In that sense, the BoxFuture approach is more flexible/easy to use, at the minor downside of adding an allocation and some dynamic function calls. (For example in both @vague's and @qui…

Where a closure can't be used but a function pointer can

[vague]

Yes, it is. Rust Playground

---

@KernelMC

I ran into a new compiler error.

error[E0644]: closure/generator type that references itself

Interestingly, the error in nightly Rust differs:

error[E0282]: type annotations needed
  --> src/lib.rs:29:35
   |
29 |     let data_len = access(&data, |message| async move { message.len() }).await;
   |                                   ^^^^^^^               ------- type must be known at this point
   |
help: consider giving this closure parameter an explicit type
   |
29 |     let data_len = access(&data, |message: /* Type */| async move { message.len() }).await;
   |                                          ++++++++++++

and if we annotate the argument, we'll get

error: lifetime may not live long enough
  --> src/lib.rs:29:50
   |
29 |     let data_len = access(&data, |message: &str| async move { message.len() }).await;
   |                                            -   - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ returning this value requires that `'1` must outlive `'2`
   |                                            |   |
   |                                            |   return type of closure `[async block@src/lib.rs:29:50: 29:78]` contains a lifetime `'2`
   |                                            let's call the lifetime of this reference `'1`

which is the same error in the first link I posted.

[zirconium-n]

I've ran into this problem. The only "real" solution when you cannot accept boxing is writing a new trait and annotate the lifetime. But with async this still require GAT+TAIT to work. It's fine if you use nightly. but if you are using stable rust only, then you have to use GAT and manually implement Future with your new type. (and manually implementing complex Future is a pain without TAIT)

[vague]

Custom closure structs work, as long as they don't capture references, like

struct Cb2(Arc<str>);
impl<'a> FnOnce<(&'a str,)> for Cb2 {
    type Output = impl 'a + Future<Output = usize>;
    extern "rust-call" fn call_once(self, args: (&'a str,)) -> Self::Output {
        async move {
           // async code
        }
    }
}

I came up with a macro for it

macro_rules! async_closure {
    ($name:ident, [$($fields:ty),+] ; [$($init:expr),+] ; $self:ident, $args:ident, $e:expr) => {{
        struct $name($($fields,)+);
        impl<'a> FnOnce<(&'a str,)> for $name {
            type Output = impl 'a + Future<Output = usize>;
            extern "rust-call" fn call_once($self, $args: (&'a str,)) -> Self::Output {
                async move { $e }
            }
        }
        $name($($init),+)
    }};
}

let cb3 = async_closure!(Cb3, [Arc<str>, usize]; [s.clone(), u]; self, args, {
    // self.n => the captured variable
    // args.n => the inputs in FnOnce
});
let data_len = access(&data, cb3).await;

Rust Playground

Note:

• several nightly features are required
• again, don't capture references (otherwise, you'll meet error: lifetime may not live long enough)
• custom types of inputs and output in FnOnce are feasible
• not polished: e.g. should use absolute paths in macros

[vague]

Improved a bit

macro_rules! async_closure {
    (
        { $( $field:ident : $t:ty  = $init:expr ),+ };
        ($( $args:ident ),+ , );
        $e:expr
    ) => {{
        struct AsyncClosure {
            $( $field: $t ),+
        }
        impl<'a> FnOnce<(&'a str,)> for AsyncClosure {
            type Output = impl 'a + Future<Output = usize>;
            extern "rust-call" fn call_once(self, args: (&'a str,)) -> Self::Output {
                let Self { $( $field ),+ } = self;
                #[allow(unused_parens)]
                let ( $( $args ),+ , ) = args;
                async move { $e }
            }
        }
        #[allow(clippy::redundant_field_names)]
        AsyncClosure{ $($field: $init),+ }
    }};
}

let cb4 = async_closure!({
    s: Arc<str> = s.clone(),
    u: u64      = u as _
}; (arg, ); { // code in async block
    // field names are names of variable
    // args names are also user-friendly
    println!("the first argument captured: {s:?}");
    println!("sleep for {u} secs");
    tokio::time::sleep(tokio::time::Duration::from_secs(u)).await;
    arg.len()
});
let _data_len = access(&data, cb4).await;

Rust Playground

[vague]

Another approach is more powerful: we have async_fn_in_trait now (in nightly Rust), which means you can just define your own async trait:

trait Accessor<'a, T> { // or trait Accessor<T> if you don't capture references
    async fn call(self, message: &str) -> T;
}

// modify the access fn's signature
async fn access<'a, T, F>(accessor: F) -> T
where
    F: Accessor<'a, T>, // 'a means the captured values
{
    let s = String::from("-");
    accessor.call(&s).await // we don't need to explicitly use HRTB any more!
}

then implement any callback struct with it

struct Cb<'s> { // capture a &str
    s: &'s str,
}
impl<'s> Accessor<'s, usize> for Cb<'s> {
    async fn call(self, message: &str) -> usize {
        ready(self.s).await;
        self.s.len() + message.len()
    }
}

struct Cb2<'s> { // capture a &str and a usize
    s: &'s str,
    u: usize,
}
impl<'s> Accessor<'s, &'s str> for Cb2<'s> {
    async fn call(self, message: &str) -> &'s str {
        ready(message.len() + self.u).await;
        self.s
    }
}

try it: Rust Playground

// Note: the result of access is still generic, exactly what you want!
async fn test_access() {
    let outer = String::from("+");

    let n = access(Cb { s: &outer }).await;
    assert_eq!(n, 2);

    let s = access(Cb2 { s: &outer, u: 123 }).await; 
    assert_eq!(s, "+");
}

It's painless, simple and elegant, and much more powerful, since you can mock FnMut and Fn just as easily as FnOnce.

• mock a capturing FnMut closure: Rust Playground
• mock a capturing Fn closure: Rust Playground

The drawback is now you must pass a value, and the callback is in the type's implementation.

[KernelMC]

For someone who's able to use the nightly compiler, I think the async_fn_in_trait solution is the best, though it does require you to define a struct which isn't super ergonomic. Unfortunately, I can't use the nightly compiler for the project I need this for.

I thought more about my design, and I've decided to no longer use closures in this API. The whole reason I was using them was to hide the lock from the caller. Instead I'm going to create new types around the lock guards and simply return those instead. Then I can hide the specific lock from the caller and avoid the many pitfalls I seem to have stumbled into.

I am still curious as to why the stable compiler claimed that the closure was capturing itself. I don't see how the closure is referencing itself, it just returns a future which takes ownership of the argument to the closure. Do I not understand how this is being compiled, or is this a bug in the compiler?

[quinedot]

@KernelMC

I am still curious as to why the stable compiler claimed that the closure was capturing itself. I don't see how the closure is referencing itself, it just returns a future which takes ownership of the argument to the closure. Do I not understand how this is being compiled, or is this a bug in the compiler?

You mean, in your OP?

async fn access<T, Fut, F>(data: &RwLock<String>, accessor: F) -> T
where
    F: FnOnce(&str) -> Fut,
    Fut: Future<Output = T>,

It's what I talk about here, Fut has to resolve to a single type, which means it can't resolve to a type that is parameterized by the lifetime of the input &str. And that's exactly what your closure is:

|message| async move { message.len() }

Because it holds on to the input message, and thus must not outlive it. It's something like a

struct AsyncClosure<'a> {
    message: &'a str,
}

(but probably more complicated to track the async generator state).

[KernelMC]

No, that error makes sense. It's the infinitely sized type error that's confusing:

error[E0644]: closure/generator type that references itself
  --> src/main.rs:23:34
   |
23 |     let data_len = access(&data, |message| async move { message.len() }).await;
   |                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cyclic type of infinite size
   |
   = note: closures cannot capture themselves or take themselves as argument;
           this error may be the result of a recent compiler bug-fix,
           see issue #46062 <https://github.com/rust-lang/rust/issues/46062>
           for more information

For more information about this error, try `rustc --explain E0644`.

The full example is on the playground.

Interestingly, the closure in question is a temporary value, so it is never even named. How then does this closure capture a reference to itself when it cannot even name itself?

[quinedot]

Looks like this issue. As @vague pointed out, it'll be a lifetime error instead once beta lands. See also the PR that changed the error.

[vague]

@vague

Another approach is more powerful: we have async_fn_in_trait now (in nightly Rust), which means you can just define your own async trait

I've used the trick to write a bunch of macros: async_closure - Rust

And the problem in this thread can be easily solved with it in nightly Rust

But I don't recommand using it: the best solution is code refactoring with a nicer design.

zjp-CN/async_closure/blob/a1f384ce3ba46faad27dc4aaeee85d9b201b5339/examples/urlo.rs#L50