Recursive async function that borrows mutable variable twice

[guerlando]

I'm designing a connection retry function. It retries if the error was ConnectionClosed. In the full implementation it has other things that justify it being a recursion, but this is off topic. Anyways, since it's a connection, it makes sense to be async. However, async recursives force me to use BoxFuture, and we know that asyncs capture and return all the arguments.

use std::future::Future;
use futures::future::{BoxFuture, FutureExt};

struct Message{}

struct Client{}

enum Error {
    ConnectionClosed
}

impl Client {
    async fn send_and_expect<'a>(
        &'a mut self,
        message: &Message
    ) -> std::result::Result<(), Error> {
        Ok(())
    }
    
    //With this function I can wrap any async function f that grabs data from the internet
    pub fn connection_retrier<'a, T>(
        f: fn(&'a mut Self, &'a Message) -> T,
        f_self: &'a mut Self,
        f_m: &'a Message,
    )-> BoxFuture<'a, std::result::Result<(),Error>>
    where
        T: Future<Output = std::result::Result<(), Error>> + 'a
    {
        async move {
            //in the original implementation this would have a delay and limited retry count
            match Client::connection_retrier(f, f_self, f_m).await {
                Ok(r) => Ok(r),
                Err(Error::ConnectionClosed) => {
                    Client::connection_retrier(f, f_self, f_m).await
                }
            }
        }.boxed()
    }
}

Error:

error[E0499]: cannot borrow `*f_self` as mutable more than once at a time
  --> src/lib.rs:32:51
   |
21 |         f: fn(&'a mut Self, &'a Message) -> T,
   |         - lifetime `'1` appears in the type of `f`
...
29 |             match Client::connection_retrier(f, f_self, f_m).await {
   |                   ------------------------------------------
   |                   |                             |
   |                   |                             first mutable borrow occurs here
   |                   argument requires that `*f_self` is borrowed for `'1`
...
32 |                     Client::connection_retrier(f, f_self, f_m).await
   |                                                   ^^^^^^ second mutable borrow occurs here

Playground

I understand why the error occurs: Client::connection_retrier(f, f_self, f_m).await, let's call it r, holds a mutable reference to f_self, so I cannot use it again while it's being held. However, after I check that this result r is Error::ConnectionClosed, I don't need it anymore, so there should be a way to discard it so I can reborrow it mutably.

[cuviper]

Your declaration forces the function to use the same 'a in both calls. If you relax that to fn(&mut Self, &Message), it works.

[guerlando]

why this works? I though the problem was that &'a mut f_self could be in the BoxFuture returned. However, the problem is solved by taking off the lifetime from &'a mut Self, &'a Message, essentially giving them other lifetimes.

[guerlando]

I made a much more detailed example:

use std::future::Future;
use futures::future::{BoxFuture, FutureExt};

struct Message{}

struct Client{}

enum Error {
    ConnectionClosed
}

impl Client {
    fn send_and_expect<'a>(
        &'a mut self,
        message: &'a Message
    ) -> BoxFuture<'a, std::result::Result<(), Error>> {
        async move {
            Ok(())
        }.boxed()
    }
    
    //With this function I can wrap any async function f that grabs data from the internet
    pub fn connection_retrier<'a, T>(
        f: fn(&'a mut Self, &'a Message) -> T,
        f_self: &'a mut Self,
        f_m: &'a Message,
    ) -> BoxFuture<'a, std::result::Result<(),Error>>
    where
        T: Future<Output = std::result::Result<(), Error>> + 'a + Send
    {
        /*
            By making f: fn(&'a mut Self, &'a Message) -> T, we tell the compiler that
            `f` is a specific function: one that specifically requires the `Self` and `Message` 
            arguments to live as long as `'a`, where `'a` is the min of the lifetimes of the `f_self` and `f_m`
            arguments passed to `connection_retrier`.
            Thus this forces `f_self` and `f_m` to live as long as `connection_retrier`.
            The call `let r = f(f_self, f_m).await` returns a `Future` that lives as long as `'a`.
            I think this is the problem. The result of the first call borrows `f_self` and `f_m`
            for at least the entire lifetime of `r`. This would make it impossible to use
            `f_self` and `f_m` again inside `connection_retrier`. As you see, I tried making sure
            that `r` destructs before we use `f_self` in `connection_retrier` again, but somehow
            `r` is marked to live as long as `connection_retrier`, because `f_self` is still considered
            to be borrowed.
        */
        async move {
            let ok: bool; 
            {
                let r = f(f_self, f_m).await;
                match r {
                    Ok(_) => ok = true,
                    Err(Error::ConnectionClosed) => {
                        ok = false;
                    }
                }
            }
            match ok {
                true => Ok(()),
                false => Client::connection_retrier(f, f_self, f_m).await
            }
        }.boxed()
    }
    
    async fn send_with_retry<'a> (
        &'a mut self,
        message: &'a Message,
    ) -> std::result::Result<(), Error>
    {
        Client::connection_retrier(
            Client::send_and_expect,
            self,
            message
        ).await
    }
}

Now I explain in the commends why your solution won't work in the real case where I was working. Could you explain my confusion cited in the commend?

Playground: Rust Playground

[cuviper]

By making f: fn(&'a mut Self, &'a Message) -> T, we tell the compiler that
f is a specific function: one that specifically requires the Self and Message
arguments to live as long as 'a, where 'a is the min of the lifetimes of the f_self and f_m
arguments passed to connection_retrier.
Thus this forces f_self and f_m to live as long as connection_retrier.
The call let r = f(f_self, f_m).await returns a Future that lives as long as 'a.
I think this is the problem. The result of the first call borrows f_self and f_m
for at least the entire lifetime of r. This would make it impossible to use
f_self and f_m again inside connection_retrier. As you see, I tried making sure
that r destructs before we use f_self in connection_retrier again, but somehow
r is marked to live as long as connection_retrier, because f_self is still considered
to be borrowed.

I think you're understanding it now -- the borrow is not just tied to the life of r, but for the entire lifetime 'a! No matter what you do to r, it will be impossible to use that reference for the rest of that lifetime. In theory, the f could have used those values in some way that depends on them living throughout 'a, so you're stuck.

In my original suggestion, making the f lifetimes anonymous meant they could work with a shorter reborrowed lifetime. I'm not sure if there's any way to do that with a lifetime that will cross await, but I don't have much experience with async/await myself.

[guerlando]

Is there a way for f to use the f_self for the lifetime 'a if it's not Send? I think it either borrows it and return immediately or send it to another thread. Maybe the compiler could be smarter and understand that if it's not Send it simply uses and returns it.

I tried even explicitly dropping r before borrowing f_self again but it doesn't work. Indeed the compiler assumes f_self is borrowed for the entire lifetime 'a.

I think the only change of solution is to make f: fn(...) not require the 'a lifetime, which makes a lot of sense. However we end up with another difficult error: Rust Playground

   Compiling playground v0.0.1 (/playground)
error[E0308]: mismatched types
  --> src/lib.rs:48:13
   |
48 |             Client::send_and_expect,
   |             ^^^^^^^^^^^^^^^^^^^^^^^ one type is more general than the other
   |
   = note: expected fn pointer `for<'r, 's> fn(&'r mut Client, &'s Message) -> Pin<Box<dyn futures::Future<Output = std::result::Result<(), Error>> + std::marker::Send>>`
              found fn pointer `for<'r, 'a> fn(&'a mut Client, &'r Message) -> Pin<Box<(dyn futures::Future<Output = std::result::Result<(), Error>> + std::marker::Send + 'a)>>`

Since send_and_expect is async, it captures the lifetime of all the references used. So the return type of send_and_expect<'a> contains the lifetime 'a.

However, connection_retrier expects a function pointer a little different than that:

pub fn connection_retrier<'a, T>(
    f: fn(&mut Self, &Message) -> T,
    f_self: &'a mut Self,
    f_m: &'a Message,
) -> BoxFuture<'a, std::result::Result<(),Error>>
where
    T: Future<Output = std::result::Result<(), Error>> + 'a + Send

I think it requires one with the specific lifetime of return type 'a. I think send_and_expect is more generic than fn because the return type has lifetime 'c: 'a + 'b, that is, a lifetime that lives as long as 'a and 'b

What you think?