Can't express lifetime for closure arguments (needed for async)

help
1

My following problem might be the same as: Lifetime may not live long enough for an async closure.

However, I'd like to present a boiled-down example and ask a few questions in that matter.

Consider the following code:

use std::future::Future;

async fn call_closure<C, Fut>(mut closure: C)
where
    C: FnMut(&str) -> Fut, // `&str` needs to outlive `Fut`, but how to say?
    Fut: Future<Output = ()>,
{
    let s = String::from("Hello World!");
    closure(&s).await;
}

#[tokio::main]
async fn main() {
    call_closure(|arg| async move {
        println!("{arg}");
    })
    .await;
}

(Playground)

Errors:

   Compiling playground v0.0.1 (/playground)
error: lifetime may not live long enough
  --> src/main.rs:14:24
   |
14 |       call_closure(|arg| async move {
   |  ___________________----_^
   | |                   |  |
   | |                   |  return type of closure `impl Future<Output = [async output]>` contains a lifetime `'2`
   | |                   has type `&'1 str`
15 | |         println!("{arg}");
16 | |     })
   | |_____^ returning this value requires that `'1` must outlive `'2`

error: could not compile `playground` due to previous error

There seems to be a workaround:

 use std::future::Future;
+use std::pin::Pin;
 
-async fn call_closure<C, Fut>(mut closure: C)
+async fn call_closure<C>(mut closure: C)
 where
-    C: FnMut(&str) -> Fut, // `&str` needs to outlive `Fut`, but how to say?
-    Fut: Future<Output = ()>,
+    C: for<'a> FnMut(&'a str) -> Pin<Box<dyn 'a + Future<Output = ()>>>,
 {
     let s = String::from("Hello World!");
     closure(&s).await;
 }
 
 #[tokio::main]
 async fn main() {
-    call_closure(|arg| async move {
-        println!("{arg}");
+    call_closure(|arg| {
+        Box::pin(async move {
+            println!("{arg}");
+        })
     })
     .await;
 }

(Playground)

My problems with this solution:

  • The syntax is kinda bloated. Particularly, I will have to add a Box::pin(async move {…}) to each invocation of the function taking the closure as argument.
  • I assume the heap allocation is unnecessary?
  • In real-world code (other than this short example), things get complex quickly and I run into error messages during compilation which are difficult to understand (and which might even involve compiler errors as I have been getting something like "expected X, found X", which I can't reproduce now, though).

My question:

Is there any progress (or existing alternative) on how to solve this problem in a clean way?

Closures are a core-feature of Rust that I often use. And since I started doing a lot of asynchronous programming, I run into problems again and again because Rust doesn't let me express lifetimes properly in an easy way. And when I use workarounds (such as the workaround above), I often end up with complex constructs and sometimes even weird compiler errors.

:sob:

All I would like to do is express lifetime relationships between arguments and the returned future of a closure. What can I do? Maybe there's some trick to use a type constructor or something? Or a feature gate that I can use?

2

There is no elegant solution, unfortunately. You can do this:

async fn call_closure<C>(mut closure: C)
where
    C: for<'a> AsyncFnMutStr<'a>,
{
    let s = String::from("Hello World!");
    closure.call(&s).await;
}

trait AsyncFnMutStr<'a> {
    type F: Future<Output = ()> + 'a;
    
    fn call(&mut self, arg: &'a str) -> Self::F;
}

impl<'a, T, F> AsyncFnMutStr<'a> for T
where
    T: FnMut(&'a str) -> F,
    F: Future<Output = ()> + 'a,
{
    type F = F;
    fn call(&mut self, arg: &'a str) -> Self::F {
        (self)(arg)
    }
}

Unfortunately the compiler will sometimes fail to infer that closures implement the trait, though it is usually able to see that an async fn pointer should implement it.

2 Likes
3

Very interesting trick!

I tried to put it to practice (to see if it helps me get around my compiler issues that I have when using the dyn approach), but I failed already at a simple playground example:

use std::future::Future;

async fn call_closure<C>(mut closure: C)
where
    C: for<'a> AsyncFnMutStr<'a>,
{
    let s = String::from("Hello World!");
    closure.call(&s).await;
}

trait AsyncFnMutStr<'a> {
    type F: Future<Output = ()> + 'a;
    
    fn call(&mut self, arg: &'a str) -> Self::F;
}

impl<'a, T, F> AsyncFnMutStr<'a> for T
where
    T: FnMut(&'a str) -> F,
    F: Future<Output = ()> + 'a,
{
    type F = F;
    fn call(&mut self, arg: &'a str) -> Self::F {
        (self)(arg)
    }
}

#[tokio::main]
async fn main() {
    call_closure(|arg| async move {
        println!("{arg}");
    })
    .await;
}

(Playground)

Errors:

   Compiling playground v0.0.1 (/playground)
error: implementation of `AsyncFnMutStr` is not general enough
  --> src/main.rs:30:5
   |
30 |     call_closure(|arg| async move {
   |     ^^^^^^^^^^^^ implementation of `AsyncFnMutStr` is not general enough
   |
   = note: `[closure@src/main.rs:30:18: 32:6]` must implement `AsyncFnMutStr<'0>`, for any lifetime `'0`...
   = note: ...but it actually implements `AsyncFnMutStr<'1>`, for some specific lifetime `'1`

error: could not compile `playground` due to previous error

Is this the fail that "sometimes" happens, or a different issue?

I would say that [closure@src/main.rs:30:18: 32:6] does implement AsyncFnMutStr<'0>, for any lifetime '0, so the error seems wrong to me.

4

Well, unfortunately, yes, that's exactly the problem I was referring to. The closure should and could implement the trait, but it doesn't because the compiler is too stupid to have to be for<'a> over all lifetimes instead of trying to pick a single specific lifetime.

5

Maybe it's possible to coerce it with a function? I will try something.

6

The only thing I've heard of working is to define real async fn rather than use closures, but I would be interested to hear if you find another way.

7

I tried to use this:

fn coerce<F, R>(f: F) -> F
where
    F: for<'a> FnMut(&'a str) -> R,
{
    f
}

But it didn't help (Playground).

8

That's too bad.

9

Ah, actually, that one definitely wont work because &'a str and &'b str are different types, but you require it to have the same return type R regardless of what the lifetime is.

2 Likes
10

I think thought I found a solution with nightly Rust using TAITs, but please correct me if there's something wrong with it:

#![feature(type_alias_impl_trait)]

use std::future::Future;

type CallClosureRetval<'a> = impl Future<Output = ()>;

async fn call_closure<C>(mut closure: C)
where
    C: for<'a> FnMut(&'a str) -> CallClosureRetval<'a>,
{
    let s = String::from("Hello World!");
    closure(&s).await;
}

#[tokio::main]
async fn main() {
    call_closure(|arg| async move {
        println!("{arg}");
    })
    .await;
}

(Playground)

Output:

Hello World!

P.S.: There is something wrong with it. It will only work with one closure (type).

The following example shows how this attempt fails:

#![feature(type_alias_impl_trait)]

use std::future::Future;

type CallClosureRetval<'a> = impl Future<Output = ()>;

async fn call_closure<C>(mut closure: C)
where
    C: for<'a> FnMut(&'a str) -> CallClosureRetval<'a>,
{
    let s = String::from("Hello World!");
    closure(&s).await;
}

#[tokio::main]
async fn main() {
    call_closure(|arg| async move {
        println!("{arg}");
    })
    .await;
    call_closure(|arg| async move {
        println!("{arg}");
    })
    .await;
}

(Playground)

Errors:

   Compiling playground v0.0.1 (/playground)
error: concrete type differs from previous defining opaque type use
  --> src/main.rs:21:18
   |
21 |       call_closure(|arg| async move {
   |  __________________^
22 | |         println!("{arg}");
23 | |     })
   | |_____^ expected `impl Future<Output = [async output]>`, got `impl Future<Output = [async output]>`
   |
note: previous use here
  --> src/main.rs:17:18
   |
17 |       call_closure(|arg| async move {
   |  __________________^
18 | |         println!("{arg}");
19 | |     })
   | |_____^

error: could not compile `playground` due to previous error

:pensive:

P.P.S.: I attempted to use a dummy type to allow for different closure types to be used, but without success:

  • Playground 1;
  • Playground 2, gives some weird "expected opaque type impl Future<Output = ()> found opaque type impl Future<Output = [async output]>" error;
  • Playground 3, crashes with rustc 1.60.0-nightly (5d8767cb2 2022-02-12) on my FreeBSD system (thread 'rustc' panicked at 'called Result::unwrap() on an Err value: UnresolvedTy(_#15t)', compiler/rustc_typeck/src/check/writeback.rs:509:75), but apparently doesn't crash Playground.
11

Question: Doesn't that mean that F (the future) lives at least as long as 'a? I would assume that is wrong because the future must not outlive 'a, right?

What we need is that the string slice lives at least as long as the future, so the future must not live longer than 'a. But if I understand right, then T: 'lt means that T lives as long as 'lt or longer. (But I always get confused on this.)

The reference says:

T: 'a means that all lifetime parameters of T outlive 'a. For example, if 'a is an unconstrained lifetime parameter, then i32: 'static and &'static str: 'a are satisfied, but Vec<&'a ()>: 'static is not.

12

It means that no lifetime annotated on the type F is shorter than 'a. Equivalently, it means that anywhere inside the region 'a, values of type F are valid (i.e. do not have dangling pointers).

13

I intuitively (wrongly) thought that this isn't what we need (regarding my original problem), because I believed:

  • &str needs to live as long or longer than the future

But, in fact, your bound in your workaround is correct. What we need is:

  • The closure passed to the function can take any &'a str as argument (i.e. with any lifetime 'a) and will return a future that lives at least as long as 'a then.

So it's alright that the future may live longer. Yet we must accept any closure where the argument &'a str uses a possibly very short lifetime 'a. The returned future must then live at least as long as this arbitrarily short lifetime 'a. I guess that's exactly what HRTBs are about. (I wrongly thought that this means &str needs to live as long or longer than the future, which isn't true.)

The underlying problem (issue 1):

Now the problem seems to be that I cannot express something like the following:

fn foo<C, Fut<'a>>(mut closure: C)
where
    C: for<'a> FnMut(&'a str) -> Fut<'a>
    Fut<'a>: 'a + Future<Output = ()>,
{
    /* … */
}

or

fn foo<C>(mut closure: C)
where
    C: for<'a, F: 'a + Future<Output = ()>> FnMut(&'a str) -> F,
{
    /* … */
}

Not sure if these two are equivalent (if they would work). Maybe one of them would be (or be related to) what's called higher-ranked traits with type variables in RFC 387 (on HRTBs)?

Is this also something like higher-kinded types?

I feel like the (main) problem is that Rust's type system lacks some features that we need, and which cannot be expressed with HRTBs as of today. Are there any plans on extending Rust in that matter? I would assume this is a very important issue due to the rising importance of async programming?

Side-quest (issue 2):

Independently of these considerations, there is the second issue where Rust doesn't recognize that a closure implements a trait T<'a> for all lifetimes 'a. (Explained in this post above.)

Compiler crash and other problems (issue 3):

In the context of finding a solution, I ran across some other weird behavior of the compiler (including the compiler panic on FreeBSD as mentioned above but also other confusing error messages when I used the dyn workaround in real-life code). I'll try to get these things sorted (and possibly file bug reports), but it's difficult to deal with so many issues at once.

Some more questions:

So I have some more questions (to everyone):

  • Regarding issue 1:
    • Is there any issue, feature request, RFC, etc. in the bug tracker regarding fixing the shortcomings of HRTBs as of today (either in the context of Futures or traits in general)?
    • Has there been any discussion on extending Rust's type system in that matter elsewhere? Maybe this topic is something that could/should be discussed on IRLO? This might not have been relevant in the past, but with async Rust, this issue seems to be of rising importance, because Future is a trait and not a type (unless we make it a trait object, which causes overhead as in other languages).
  • Regading issue 2:
    • Is this a bug or a rather a missing feature?
    • Is there any issue in the bug tracker on this?
14

I think the above quoted syntax would not describe what I need, because F needs to be a fixed type (i.e. a type parameter to foo/call_closure) except being dependent on a lifetime that it is not fixed.

I would thus conclude that higher-kinded types (in the general meaning) are not needed to solve the problem with async closures.

The first syntax (which also doesn't exist as of today) would make more sense:

fn foo<C, Fut<'a>>(mut closure: C)
where
    C: for<'a> FnMut(&'a str) -> Fut<'a>
    Fut<'a>: 'a + Future<Output = ()>,
{
    /* … */
}

Has there been any proposal to allow something like this?

15

I don't keep up with proposals to fix this kind of stuff unfortunately.

16

Right, the whole point of TAIT is that the type alias stands for one type throughout the whole program (just like return-position impl Trait stands in for a single concrete type); you can't use the alias as an erased type that unifies several concrete types, that's what enum and dyn Trait are for.

17

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.