How to bound FnMut where output has lifetime of args (but neither are references)

help
1 
trait HasGAT {
    type Reader<'a>;

    fn get(&self) -> Self::Reader<'_>;
}

trait Read {
    fn get_u8(&self) -> &u8;
}

struct HasRef<'a, T> {
    value: &'a T,
}

fn for_each<T: HasGAT, F>(iter: impl Iterator<Item = T>, op: F)
where
    F: for<'a> FnMut(T::Reader<'a>) -> HasRef<'a, u8>,
{
    iter.map(|item| op(item.get()))
}

playground

I've found seemingly related issues like Lifetime bounds to use for Future that isn't supposed to outlive calling scope and How to Create `Fn` Bound That Includes Arguments With Higher-ranked Lifetime Bound but they always use a reference in either the input or the output of the function. In my case there are no references, rather the input type carries a lifetime.

The actual motivating case is a bit more complex and has the closure return a future:

trait HasGAT {
    type Reader<'a>
    where
        Self: 'a;

    fn get(&self) -> Self::Reader<'_>;
}

trait Read {
    fn get_u8(&self) -> &u8;
}

struct HasRef<'a, T> {
    value: &'a T,
}

async fn for_each<T: HasGAT, F, Fut>(iter: impl Iterator<Item = T>, mut op: F)
where
    F: for<'a> FnMut(T::Reader<'a>) -> Fut,
    Fut: futures::future::Future<Output = ()>,
{
    use futures::stream::StreamExt as _;

    futures::stream::iter(iter)
        .for_each(|item| op(item.get()))
        .await
}

async fn use_for_each<T: HasGAT + 'static>(iter: impl Iterator<Item = T>)
where
    for<'a> T::Reader<'a>: Read,
{
    for_each(iter, |item| async move {
        dbg!(*item.get_u8());
    }).await
}

playground

error: lifetime may not live long enough
  --> src/lib.rs:33:27
   |
33 |       for_each(iter, |item| async move {
   |  _____________________-----_^
   | |                     |   |
   | |                     |   return type of closure `[async block@src/lib.rs:33:27: 35:6]` contains a lifetime `'2`
   | |                     has type `<T as HasGAT>::Reader<'1>`
34 | |         dbg!(*item.get_u8());
35 | |     }).await
   | |_____^ returning this value requires that `'1` must outlive `'2`

error: could not compile `playground` (lib) due to previous error

Tagging some people who might know things: @quinedot, @Yandros @vague

EDIT: this is the same as How to express that the Future returned by a closure lives only as long as its argument? except that the argument isn't a reference.

2

I'll read your specific case deeper when I get a moment, but here's where I often end up pointing people. It was written pre-GAT and some of the lifetime-parameterized traits could probably move the parameter to a GAT.

The core issue is that Fut can't be a type parameter, because type parameters resolve to a single type, whereas you need a type constructor (something parameterized by a lifetime -- a different type per lifetime). Since you don't have a concrete type constructor like &_ handy, you need to emulate one via a trait.

1 Like
3

Something like this for the signature issue, but now there are new errors about returning local borrows and (I think) futures holding onto local borrows.[1]

  1. And inference took a hit as per normal with this sort of fix. ↩ī¸Ž

4

there's trick to use parameterized return type instead of associated type with a sub-trait , similar to this:

trait FnMut1WithRet<Ret, Arg1>: FnMut(Arg1) -> Ret {}
impl<F, Arg1, Ret> FnMut1WithRet<Ret, Arg1> for F where F: FnMut(Arg1) -> Ret {}

then your first example almost compiles, but since the iterator yields T by value, the closure passed to map cannot borrow it. you need to change the iterator to yield reference, and then it compiles.

I believe you can use async-fn-traits for the async case, which uses the same sub-trait trick.

2 Likes
6

The problem with this code is that the closure passed to map/for_each takes ownership of a T, then creates a T::Reader<'a> that borrows from that T and finally calls op which creates a HasRef<'a> that still borrows from that T. This is finally returned by the closure, but at that point theT goes out of scope, gets dropped, and hence the HasRef<'a> becomes dangling. Hence you will never to express this through types alone, you need to change the semantics, e.g. by passing ownership of T to op, or by

2 Likes
7

Yeah, that example is bad. The async example is more in line with what I'm actually trying to do, which is to have the async closure take ownership of the HasRef<'_> but otherwise not return anything derived from it.

8

This doesn't address the async case - the problem in that case is that the lifetime of the returned future doesn't appear in any types, so it isn't possible to write F: for<'a> FnMut1WithRet<HasRef<'a, u8>, T::Reader<'a>>,.

9

In the async case Fut cannot reference the 'a lifetime due to being defined outside the scope of the for<'a>. You can fix this, but then you'll be back at the HasRef<'a> case.

10

The local borrow error is fixed by interposing an async block (playground) but then we're back to the original problem:

error: lifetime may not live long enough
  --> src/lib.rs:52:42
   |
52 |       for_each(iter, |item: T::Reader<'_>| async move {
   |  _____________________________________--_-_^
   | |                                     |  |
   | |                                     |  return type of closure `[async block@src/lib.rs:52:42: 54:6]` contains a lifetime `'2`
   | |                                     let's call the lifetime of this reference `'1`
53 | |         dbg!(*item.get_u8());
54 | |     })
   | |_____^ returning this value requires that `'1` must outlive `'2`

error: could not compile `playground` (lib) due to previous error

The issue is that this still doesn't tie the lifetime of the returned Fut to the lifetime of the argument.

11

you are right. in the async case, the return type is not a concrete type, but a opaque type which you cannot name. I think in its current shape, the only way to convey the lifetime relation is to use trait object Box<dyn Trait + 'lifetime>, e.g. the async case could be something similar to this:

F: for<'a> FnMut1WithRet<Pin<Box<dyn Future<Output = HasRef<'a, u8>> + 'a>>, T::Reader<'a>>,

then in the closure, you'll need to box the async block.

12

This works. The key trick here is promoting FnMut::Output from an associated type to a type parameter to avoid E0582:

// error[E0582]: binding for associated type `Output` references lifetime `'a`, which does not appear in the trait input types
//     |
//     |     F: Copy + for<'a> FnMut(T::Reader<'a>) -> futures::future::LocalBoxFuture<'a, Result<(), capnp::Error>>,

But I'm still not satisfied. Can we somehow avoid the boxing? I think @vague has previously attempted similar things.

13

Is there any reason to not pass the ownership in the callback? This just works

trait Read {
    fn get_u8(&self) -> &u8;
}

async fn for_each<T, F, Fut>(iter: impl Iterator<Item = T>, mut op: F)
where
    F: FnMut(T) -> Fut,
    Fut: futures::future::Future<Output = ()>,
{
    use futures::stream::StreamExt as _;
    futures::stream::iter(iter).for_each(|item| op(item)).await
}

async fn use_for_each<T: Read>(iter: impl Iterator<Item = T>) {
    for_each(iter, |item| async move {
        dbg!(*item.get_u8());
    })
    .await
}
14

I can't think of a way to do it without boxing. currently in rust, a generic parameter can only have a kind of Lifetime or Type, but we need it to be Lifetime -> Type. I've seen people using hypothetical syntax in pseudo code like this:

// WARNING: invalid rust code, for illustration only
async fn for_each<T: HasGAT, F, Fut<'_>>(iter: impl Iterator<Item = T>, mut op: F)
where
    F: for<'a> FnMut(T::Reader<'a>) -> Fut<'a>,
    for <'a> Fut<'a>: futures::future::Future<Output = ()> + 'a,
{
    //...
}

in current type system, a boxed trait object is the closest thing we get.

15

Yes, this is a simplified example. In actuality get is a method that returns an iterator of T::Reader<'_>, which I pass to stream::iter. I could move all the handling of the iterator into the async callback, but then we've needlessly increased the amount of boilerplate.

16

Is there an issue or something that you can point me to? I'd like to leave an authoritative reference in my code.

Thanks for your help!

17

Aside from the stable solutions you linked, I recommend the async fn in trait that will promisingly be stablized in next months and has landed in nightly without feature gate. AFIT almost solved the non-boxing return Future problem in any aspect.

trait AsyncCallback<T> {
    async fn call(&self, _: T);
}

Rust Playground for your case (or Rust Playground for simplicity)

it basically moves the callback (both from functions and capturing closures) into an implementation of a custom type (i.e a struct with potential captured variables as its fields which is needless for your current case since no captured variables is shown for now).

The downside is it's boring to define these types, but the boilerplate is solvable via macros. Like what I did in async_closure - Rust (Note this crate is not maintained for now because AFIT has changed a lot since I write it a half year ago)

2 Likes
18

You can have this:

Yielding:

async fn for_each<T: HasGAT, F, Fut : ForLt>(iter: impl Iterator<Item = T>, mut op: F)
where
    F: for<'a> FnMut(T::Reader<'a>) -> Fut::Of<'a>,
    for<'a> Fut::Of<'a>: futures::future::Future<Output = ()> + 'a,
{
    //...
}

The drawback is that these "For types" do not play well with inference, so you have to name them:

  • e.g.,

    for_each::<_, _, ForLt!(LocalBoxFuture<'_, _>)>(...)

The main tension here anyways will be that:

  • Futures being anonymous types, naming them is hard, and it's best left for type inference to figure out.

  • |...| ... literal closure expressions, won't get properly for<'any>-promoted unless there are very concrete type-constraints (properly lifetime-infected) beforehand.

When you are dealing with a pure async fn definition, these already get the proper for<'any>-signature you may need, and in that case you do not need the latter point, so that you can just make do with "inference". In that case, something like https://docs.rs/async-fn-traits is quite handy for this.

Otherwise, you'll need to name the type, which means:

  • {Local,}BoxFuture<'lt, _> or variants thereof;

  • or nightly-only type_alias_impl_trait, with which you can suddenly "name" things such as these anonymous futures.

3 Likes