Lifetime for (async) closure args / Problem with workaround: expected X found (same) X

Continuing the discussion from Can't express lifetime for closure arguments (needed for async):

I still would like to avoid the workaround (as discussed in the original thread). However, until there's a solution, I'm willing to use Pin<Box<dyn Future>>.

The playground linked above does work. However, when things get a bit more complex, I run into problems:

I was able to reproduce the error:

use std::future::Future;
use std::pin::Pin;

trait Something {
    fn something(&self) {}
}

struct Executor<I> {
    inner: I,
}

async fn call_closure<C>(mut closure: C)
where
    C: for<'a> FnMut(&'a str) -> Pin<Box<dyn 'a + Future<Output = ()>>>,
{
    let s = String::from("Hello World!");
    closure(&s).await;
}

impl<I> Executor<I>
where
    I: Something,
{
    pub async fn run(&self) {
        call_closure(|s| {
            Box::pin(async move {
                self.inner.something();
                println!("{s}");
            })
        })
        .await;
    }
}

(Playground)

Errors:

   Compiling playground v0.0.1 (/playground)
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
  --> src/lib.rs:24:22
   |
24 |     pub async fn run(&self) {
   |                      ^^^^^
   |
note: first, the lifetime cannot outlive the lifetime `'_` as defined here...
  --> src/lib.rs:24:22
   |
24 |     pub async fn run(&self) {
   |                      ^
note: ...so that the expression is assignable
  --> src/lib.rs:24:22
   |
24 |     pub async fn run(&self) {
   |                      ^^^^^
   = note: expected `&Executor<I>`
              found `&Executor<I>`
note: but, the lifetime must be valid for the anonymous lifetime #1 defined here...
  --> src/lib.rs:25:22
   |
25 |           call_closure(|s| {
   |  ______________________^
26 | |             Box::pin(async move {
27 | |                 self.inner.something();
28 | |                 println!("{s}");
29 | |             })
30 | |         })
   | |_________^
note: ...so that the types are compatible
  --> src/lib.rs:26:13
   |
26 | /             Box::pin(async move {
27 | |                 self.inner.something();
28 | |                 println!("{s}");
29 | |             })
   | |______________^
   = note: expected `Pin<Box<dyn Future<Output = ()>>>`
              found `Pin<Box<dyn Future<Output = ()>>>`

For more information about this error, try `rustc --explain E0495`.
error: could not compile `playground` due to previous error

I don't understand the error message:

error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
  --> src/lib.rs:24:22
   |
24 |     pub async fn run(&self) {
   |                      ^^^^^
   |
note: first, the lifetime cannot outlive the lifetime `'_` as defined here...
  --> src/lib.rs:24:22
   |
24 |     pub async fn run(&self) {
   |                      ^

self needs to outlive itself!?

   = note: expected `&Executor<I>`
              found `&Executor<I>`

Uhhhh!?

   = note: expected `Pin<Box<dyn Future<Output = ()>>>`
              found `Pin<Box<dyn Future<Output = ()>>>`

:crazy_face:

This doesn't make sense (to me). Any idea what's going on or wrong in my code?

1 Like

That's because the future contains a reference to self, but if you call the closure with 'a set to 'static then that wouldn't work since the reference to self is not valid everywhere inside the region 'static.

1 Like

Do you mean the Box::pinned future? Or the future returned by async fn run?

I mean the future you put into the box. This one:

            Box::pin(async move {
                self.inner.something();
                println!("{s}");
            })
1 Like

Okay, thanks a lot. I think I understand the problem now. (Though I think the compiler's error message is a bit weird.)

Unfortunately, I don't see a solution to this problem. Is there any?

I guess I'd need to restrict 'a to not be any lifetime, but only any lifetime shorter than some other lifetime 'b that I would pass as lifetime argumdent to call_closure.

Something like:

-async fn call_closure<C>(mut closure: C)
+async fn call_closure<'b, C>(mut closure: C)
 where
     C: for<'a> FnMut(&'a str) -> Pin<Box<dyn 'a + Future<Output = ()>>>,
+    // restrict 'a such that 'b: 'a, but how?
 {


I assume the only solution is to avoid using references at all?

Like this:

use std::future::Future;
use std::rc::Rc;

trait Something {
    fn something(&self) {}
}

struct Executor<I> {
    inner: I,
}

async fn call_closure<C, R>(mut closure: C)
where
    C: FnMut(Rc<String>) -> R,
    R: Future<Output = ()>,
{
    let s = Rc::new(String::from("Hello World!"));
    closure(s.clone()).await;
}

impl<I> Executor<I>
where
    I: Something,
{
    pub async fn run(&self) {
        call_closure(|s| {
            Box::pin(async move {
                self.inner.something();
                println!("{s}");
            })
        })
        .await;
    }
}

#[tokio::main]
async fn main() {
    struct S {}
    impl Something for S {
        fn something(&self) {}
    }
    let executor = Executor { inner: S {} };
    executor.run().await;
}

(Playground)

Output:

Hello World!

I don't really like this, because all this seems just due to a lack of expressiveness regarding lifetimes. Is there any other way that doesn't make me clone the String? (In my real life code, it's not a String but a more complex structure.)

I don't know.

Okay, thanks anyway very much for your help with this (and the other) issue. At least I have an idea now what will not work :wink: (and why it doesn't work).

It is actually possible to restrict lifetimes in the way you want. I read about this at Closure lifetime question and at Argument requires that ... is borrowed for 'static.

You would change your signature to:

async fn call_closure<'max, C>(mut closure: C)
where
    C: for<'a> FnMut(&'a str, [&'a &'max (); 0]) -> Pin<Box<dyn 'a + Future<Output = ()>>>

and the [&'a &'max (); 0] argument would generate an implicit bound that requires 'a to be shorter than 'max.

(Playground)

4 Likes

Very nice! Maybe using a PhantomData would be more idiomatic though? I tried it. Seems to work. (Playground)

It's still a bit noisy with the extra argument, so not sure if I'd want to use it in a public interface, but at least it's possible.


You can also avoid the extra argument and modify the return value instead:

use std::future::Future;
use std::marker::PhantomData;
use std::pin::Pin;

struct DynFutureWithLt<'max, 'min, O> {
    future: Pin<Box<dyn 'min + Future<Output = O>>>,
    phantom: PhantomData<&'min &'max ()>,
}

impl<'max, 'min, O> DynFutureWithLt<'max, 'min, O> {
    fn new(future: Pin<Box<dyn 'min + Future<Output = O>>>) -> Self {
        Self {
            future,
            phantom: PhantomData,
        }
    }
}

async fn call_closure<'max, C>(mut closure: C)
where
    C: for<'a> FnMut(&'a str) -> DynFutureWithLt<'max, 'a, ()>,
{
    let s = String::from("Hello World!");
    closure(&s).future.await;
}

trait Something {
    fn something(&self) {}
}

struct Executor<I> {
    inner: I,
}

impl<I> Executor<I>
where
    I: Something,
{
    pub async fn run<'a>(&'a self) {
        call_closure::<'a, _>(|s| {
            DynFutureWithLt::new(Box::pin(async move {
                self.inner.something();
                println!("{s}");
            }))
        })
        .await;
    }
}

#[tokio::main]
async fn main() {
    struct S;
    impl Something for S {}
    Executor { inner: S }.run().await;
}

(Playground)

Output:

Hello World!

1 Like

We can do even better and avoid both the extra argument and the newtype pattern for the return value of the closure (and use an ordinary Pin<Box<Future>> instead).

This is how it works:

pub trait WithPhantom<P: ?Sized> {}
impl<T: ?Sized, P: ?Sized> WithPhantom<P> for T {}

pub trait FutureWithPhantom<O, P: ?Sized>:
    Future<Output = O> + WithPhantom<P> {}

impl<O, P: ?Sized, T> FutureWithPhantom<O, P> for T where
    T: Future<Output = O> {}

async fn call_closure<'max, C>(mut closure: C)
where
    C: for<'a> FnMut(
        &'a str,
    ) -> Pin<
        Box<dyn 'a + FutureWithPhantom<(), PhantomData<&'a &'max ()>>>,
    >,
{
    let s = String::from("Hello World!");
    closure(&s).await;
}

Then we can invoke it with:

call_closure::<'a, _>(|s| Box::pin(async move { /* … */ }))

No dummy arguments, and we can use an ordinary Box::pin. :smiley:

(Playground)


P.S.: If WithPhantom<P: ?Sized> was an auto-trait, we could simply write dyn 'a + Future<…> + WithPhantom<…>.

2 Likes

The trait idea is quite neat :ok_hand:, although it would break for callers used to writing .boxed() or .boxed_local() over Box::pin. Hence why I personally find that that "discarded argument" approach with a PhantomData or an empty array can be less surprising.

As was written in the linked post, yeah you could even have a PhantomBound or something to express the intent even clearer.

In general, PhantomData<T> is a bit superior to [T; 0] insofar it doesn't raise the alignment of the data structure. But for a zero-sized parameter passed by value, the alignment doesn't matter, and writing [] is easier than writing PhantomData, hence my going for [], here.

Finally, in your case, in order to improve the ergonomics, you could instead wrap the parameter s, and use, for instance, a &'a &'max mut str[1], or some &'a Str<'max> for some helper type Str<'max> that is Deref<Target = str>. I still think I'd just go for the discarded parameter.


  1. It is sound to transmute &'a &'a T to &'a &'_ mut T ↩︎

1 Like

Ohhh, I didn't realize that. I guess that's because Box::pin returns a Pin<Box<T>> with some opaque future (T), which then converts to my FutureWithPhantom through an unsized coercion.

future::futures::FutureExt::boxed, in contrast, returns already a Pin<Box<dyn Future + Send>>, which comes with a particular vtable for the trait object (that could be 100% compatible to a FutureWithPhantom + Send, but the compiler won't notice/know/guarantee and treats these two types as distinct).

I guess if .boxed() would return a Pin<Box<Self>> instead, the problem would be solved? Like this:

/* … */

trait MyFutureExt: Future {
    fn boxed(self) -> Pin<Box<Self>>
    where
        Self: Sized,
    {
        Box::pin(self)
    }
}

impl<T: Future> MyFutureExt for T {}

/* … */

impl<I> Executor<I>
where
    I: Something,
{
    pub async fn run<'a>(&'a self) {
        call_closure::<'a, _>(|s| {
            async move {
                self.inner.something();
                println!("{s}");
            }.boxed()
        })
        .await;
    }
}

/* … */

(Playground)

Question: Do you know why .boxed() returns a Pin<Box<dyn …>> instead of Pin<Box<Self>>?

Ah, I didn't read it all.

I guess it's a matter of preference… I find the [… (); 0] a bit cryptic, but arguably it's shorter:

  • […; 0] vs PhantomData<…> in the signature
  • [] vs PhantomData when calling the closure

So maybe [T; 0] is better. Both aren't really beautiful/concise though, I think :wink:

I guess ultimately, Rust lacks some expressiveness regarding lifetime bounds (which is a problem that pops up particularly when doing async programming, but maybe also in other contexts?). See also my original thread (particularly this post and the follow-up) where I asked for help to find a solution without involving dyn at all.

:scream:

That might be even more cryptic to the reader :wink:. But perhaps I'm just not used to these tricks enough to dare to use them in my code. It would be nicer if Rust provided easier ways to express them. But I guess it's not easy to come up with a nice syntax (not to speak of the implementation). I guess there are many other more important open issues yet.

Because a Pin<Box<dyn ...>> is generally what you want when you aren't doing weird tricks like you are in this thread. Sometimes the compiler fails to insert the unsizing coercion even when you want it.

1 Like

Flexibility is a double-edged sword, given inference errors.

Imagine having a Result<impl Future, …> and wanting a Result<BoxFuture…, …>. Currently you can do:

fut_res.map(|fut| fut.boxed())
// or even, if you fancy eta reduction:
fut_res.map(FutureExt::boxed)

which, with the more general Pin<Box<Self>> abstraction, would fail.

On the other hand, the advantage of doing that is that .boxed_local() would not be needed, since your general .boxed() would handle them both :person_shrugging:


Yes, :100:%: ideally it should be possible to write for<'a where 'up : 'a> kind of bounds, at the very least.

Or to maybe make it more math-y

we could try to define "lifetime sets":

lifetimeset LessThan<'m> = for<'a> where 'm : 'a;

and then be able to write for<'a in LessThan<'m>>.

Or even maybe something with a reduced scope:

for<#[small] 'a>

Where #[small] would be a quantification wherein 'a would be guaranteed to be smaller than any other (non-higher-order) region in scope.

Hmmm, I see, so giving it an explicit type is an advantage and sometimes needed (or at least handy).

This would solve the issue with trait objects, but am I right that this wouldn't help me to solve the original problem in the other thread, i.e. where I want(ed) to avoid using dyn completely?


P.S.: I think Pin<Box<dyn Future…>> is very common in real-life code (so I shouldn't feel bad to use it), but I think it's always still a workaround more or less when using it here, and not really needed in an ideal (programming language) world.

I made another attempt to be able to use .boxed() or .boxed_local(). (Playground)

I run into error: implementation of 'MyClosure' is not general enough. Maybe the same "bug" as I had here.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.