How to write a helper function to retry a future in a loop?

I have a struct Foo with an async method which may fail (returning a bool). Therefore, I want to retry this operation() in a loop until it succeeds. There's some extra boilerplate (e.g. throttling retries) which isn't relevant here, so let's assume a simple retry loop {}.

If I directly call self.operation().await in a loop it works fine. But I'd like to introduce a helper function, generic over the exact operation, to implement the retry loop (and the throttling boilerplate). This would avoid copy-pasting the retry loop, as in practice I have various operations to retry.

However, I couldn't figure out how to precisely declare this helper function in terms of types and lifetimes to get it to compile.

What I've tried so far:

use std::future::Future;

struct Foo {}

impl Foo {
    // Operation that may fail and should be retried.
    async fn operation(&mut self) -> bool {
        unimplemented!()
    }

    // Directly retry the operation in a loop.
    async fn direct(&mut self) {
        loop {
            if self.operation().await {
                return;
            }
        }
    }

    // Retry the operation via a retry_loop helper function.
    async fn indirect_1(&mut self) {
        self.retry_loop_1(Foo::operation).await
    }

    async fn indirect_2(&mut self) {
        self.retry_loop_1(|this| this.operation()).await
    }

    // First attempt, without lifetimes.
    async fn retry_loop_1<F, Fut>(&mut self, mut f: F)
    where
        F: FnMut(&mut Self) -> Fut,
        Fut: Future<Output = bool>,
    {
        loop {
            if f(self).await {
                return;
            }
        }
    }

    async fn indirect_3(&mut self) {
        self.retry_loop_2(|this| this.operation()).await
    }

    // Second attempt, with lifetimes.
    async fn retry_loop_2<'a, 'b, F, Fut>(&'a mut self, mut f: F)
    where
        'a: 'b,
        F: FnMut(&'b mut Foo) -> Fut,
        Fut: Future<Output = bool> + 'b,
    {
        loop {
            if f(self).await {
                return;
            }
        }
    }
}

And the compiler errors:

error: implementation of `std::ops::FnOnce` is not general enough
   --> src/foo.rs:22:14
    |
22  |           self.retry_loop_1(Foo::operation).await
    |                ^^^^^^^^^^^^ implementation of `std::ops::FnOnce` is not general enough
    |
    = note: `std::ops::FnOnce<(&'0 mut foo::Foo,)>` would have to be implemented for the type `for<'_> fn(&mut foo::Foo) -> impl futures::Future {foo::Foo::operation}`, for some specific lifetime `'0`...
    = note: ...but `std::ops::FnOnce<(&mut foo::Foo,)>` is actually implemented for the type `for<'_> fn(&mut foo::Foo) -> impl futures::Future {foo::Foo::operation}`

error: lifetime may not live long enough
  --> src/foo.rs:26:34
   |
26 |         self.retry_loop_1(|this| this.operation()).await
   |                            ----- ^^^^^^^^^^^^^^^^ returning this value requires that `'1` must outlive `'2`
   |                            |   |
   |                            |   return type of closure is impl futures::Future
   |                            has type `&'1 mut foo::Foo`

error[E0499]: cannot borrow `*self` as mutable more than once at a time
  --> src/foo.rs:54:18
   |
47 |     async fn retry_loop_2<'a, 'b, F, Fut>(&'a mut self, mut f: F)
   |                               -- lifetime `'b` defined here
...
54 |             if f(self).await {
   |                --^^^^-
   |                | |
   |                | mutable borrow starts here in previous iteration of loop
   |                argument requires that `*self` is borrowed for `'b`

Note: I've here tried to minimize the problem - for more context the full code I want to refactor is in this code on GitHub.

I've found a similar question in Issue with FnOnce and async function with a &reference type argument, and managed to apply it to this simplified example, using the nightly type_alias_impl_trait feature.

#![feature(type_alias_impl_trait)]

use std::future::Future;

struct Foo {}

type FutBool<'a> = impl 'a + Future<Output = bool>;

impl Foo {
    // Operation that may fail and should be retried.
    async fn operation(&mut self) -> bool {
        unimplemented!()
    }

    async fn indirect(&mut self) {
        self.retry_loop(|this| this.operation()).await
    }

    async fn retry_loop<F>(&mut self, mut f: F)
    where
        for<'any> F: FnMut(&'any mut Foo) -> FutBool<'any>,
    {
        loop {
            if f(self).await {
                return;
            }
        }
    }
}

However, one thing to note (which is however non-blocking) is that I couldn't make a type alias generic over the future's Output.

type Fut<'a, T> = impl 'a + Future<Output = T>;
type FutBool<'a> = Fut<'a, bool>;
error[E0271]: type mismatch resolving `<impl futures::Future as futures::Future>::Output == T`
 --> src/foo.rs:5:19
  |
5 | type Fut<'a, T> = impl 'a + Future<Output = T>;
  |              -    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `bool`, found type parameter `T`
  |              |
  |              this type parameter
  |
  = note:        expected type `bool`
          found type parameter `T`

Now, this was a simplified example, in reality the type Foo is rather a Bar<'a> which contains a reference. I'm now blocked at the following stage.

struct Bar<'a> {
    data: &'a str,
}

type FutBool<'x> = impl 'x + Future<Output = bool>;

impl<'a> Bar<'a> {
    async fn indirect_4(&mut self) {
        self.retry_loop_4(|this| this.operation()).await
    }

    async fn retry_loop_4<F>(&mut self, mut f: F)
    where
        for<'any> F: FnMut(&'any mut Bar) -> FutBool<'any>,
    {
        loop {
            if f(self).await {
                return;
            }
        }
    }

    async fn operation(&mut self) -> bool {
        unimplemented!()
    }
}
error[E0700]: hidden type for `impl Trait` captures lifetime that does not appear in bounds
  --> src/bar.rs:11:34
   |
11 |         self.retry_loop_4(|this| this.operation()).await
   |                                  ^
   |
note: hidden type `impl futures::Future` captures the anonymous lifetime #2 defined on the body at 11:27
  --> src/bar.rs:11:27
   |
11 |         self.retry_loop_4(|this| this.operation()).await
   |                           ^^^^^^^^^^^^^^^^^^^^^^^

error[E0477]: the type `impl futures::Future` does not fulfill the required lifetime
 --> src/bar.rs:7:20
  |
7 | type FutBool<'x> = impl 'x + Future<Output = bool>;
  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
note: type must outlive the lifetime `'x` as defined on the item at 7:14
 --> src/bar.rs:7:14
  |
7 | type FutBool<'x> = impl 'x + Future<Output = bool>;

The way this can currently be made working is by introducing a new trait. Adapting one that I have previously presented in a different thread:

use std::future::Future;

trait AsyncSingleArgFnMut<Arg>: FnMut(Arg) -> <Self as AsyncSingleArgFnMut<Arg>>::Fut {
    type Fut: Future<Output=<Self as AsyncSingleArgFnMut<Arg>>::Output>;
    type Output;
}

impl<Arg, F, Fut> AsyncSingleArgFnMut<Arg> for F
where
    F: FnMut(Arg) -> Fut,
    Fut: Future,
{
    type Fut = Fut;
    type Output = Fut::Output;
}

struct Foo {}

impl Foo {
    // Operation that may fail and should be retried.
    async fn operation(&mut self) -> bool {
        unimplemented!()
    }

    // Directly retry the operation in a loop.
    async fn direct(&mut self) {
        loop {
            if self.operation().await {
                return;
            }
        }
    }


    // Retry the operation via a retry_loop helper function.
    async fn indirect_1(&mut self) {
        self.retry_loop_1(Foo::operation).await
    }

    // I have no idea why this still doesn’t work
    async fn indirect_2(&mut self) {
        //self.retry_loop_1(|this: &mut Self| this.operation()).await
    }

    async fn retry_loop_1<F>(&mut self, mut f: F)
    where
        for<'a> F: AsyncSingleArgFnMut<&'a mut Self, Output=bool>
    {
        loop {
            if f(self).await {
                return;
            }
        }
    }

}

As noted above, it is a bit puzzling to me why indirect_2 still doesn’t work and I have no idea how to fix it.

Edit: Here’s the adaptation to your Bar<'a> example (click to expand)

struct Bar<'a> {
    data: &'a str,
}

impl<'a> Bar<'a> {
    async fn indirect_4(&mut self) {
        self.retry_loop_4(Self::operation).await
    }

    async fn retry_loop_4<F>(&mut self, mut f: F)
    where
        for<'b> F: AsyncSingleArgFnMut<&'b mut Self, Output=bool>,
    {
        loop {
            if f(self).await {
                return;
            }
        }
    }

    async fn operation(&mut self) -> bool {
        unimplemented!()
    }
}

Thanks a lot!

This approach indeed worked in the simple case where the operation doesn't take any extra parameter: https://github.com/gendx/connect-box/commit/b5a114148a9e2b21481016187d854a617f4f593d.

However, it doesn't work in the case where a lambda has to capture a parameter, e.g.

struct Bar<'a> {
    data: &'a str,
}

impl<'a> Bar<'a> {
    async fn indirect(&mut self, param: usize) {
        // Here we cannot use `Bar::operation` to capture `param`.
        self.retry_loop(|this| this.operation(param)).await
    }

    async fn operation(&mut self, param: usize) -> bool {
        unimplemented!()
    }
}

Yes, I know! In case I find a better way, I'll tell you. It does for sure appear like Rustʼs type checker has to improve significantly around this kind of setting though, anyways. Even for the non-lambda case, defining an extra trait like this schuld IMO not be necessary.

1 Like