Mutable borrow in async function

1

I have an async function A which receives another async function B as an argument. Inside the function A, I wish to call the function B two times, passing to it a mutable borrow to some value. But the compiler complains that at the time of the second call the value is still borrowed.

Why doesn't this work? Isn't the scope of the future supposed to end after the await? Then the borrow should finish, right? What am I missing here?

Here is a minimal "toy version" of my code:

use std::future::Future;

async fn do_thing_single_value<'a, 'b, F, Fut>(value: &'a mut String, f: F) -> i32
where
    F: Fn(&'b mut String) -> Fut,
    Fut: Future<Output = i32>,
    'a: 'b,
{
    let v1: i32 = f(value).await;

    // Error here, is says value is already borrowed!
    let v2: i32 = f(value).await;

    v1 + v2
}

async fn do_thing_all_values<'a, F, Fut>(values: &'a mut Vec<String>, f: F) -> i32
where
    F: Fn(&'a mut String) -> Fut,
    Fut: Future<Output = i32>,
{
    let mut total = 0;
    for v in values.iter_mut() {
        let result = do_thing_single_value(v, &f).await;
        total += result;
    }
    total
}

async fn thing_to_do(v: &mut String) -> i32 {
    v.make_ascii_uppercase();
    v.chars()
        .map(|c| if c.is_ascii_digit() { 1 } else { 0 })
        .sum()
}

#[tokio::main]
async fn main() -> Result<(), ()> {
    let mut values = vec![String::from("asdf"), String::from("1234")];
    let result = do_thing_all_values(&mut values, thing_to_do).await;
    println!("result={:?}", result);

    Ok(())
}
2

There are several things you need to know to understand what's going on here. First of all, remember that mutable borrows must not overlap. As a consequence of this, whenever you pass a mutable reference to a function, you're actually passing a different one (as the type system understands it) — a “reborrow” of the original. The code

fn foo(input: &mut String) {
    bar(input);
    bar(input);
}

is treated as if you wrote

fn foo(input: &mut String) {
    bar(&mut *input);
    bar(&mut *input);
}

creating two mutable references with distinct, non-overlapping lifetimes. Otherwise, using a mutable reference more than once would be prohibited unless you added explicit &mut *... reborrows.

This then causes trouble with do_thing_single_value, because the lifetime of value as passed to f(value) is a lifetime that is shorter than the call to do_thing_single_value() as a whole — but the required input is &'b mut String, and 'b is a lifetime parameter, meaning the caller gets to (implicitly) pick it and it has one lifetime-value for the entire call!

In order to solve this for ordinary functions, you would use a (possibly implicit) HRTB to specify that F works for any lifetime:

async fn do_thing_single_value<'a, F>(value: &'a mut String, f: F) -> i32
where
    F: for<'b> Fn(&'b mut String) -> i32,

However, this is not sufficient for async functions, because there is also the future type. You cannot specify the future type as a type parameter, because the future captures the mutable reference, so the future type depends on the lifetime.

My favorite way to solve this problem is the tiny async_fn_traits library, which lets you avoid mentioning the future type at all if you don't need to. This version of your code compiles:

use async_fn_traits::AsyncFn1;

async fn do_thing_single_value<'a, F>(value: &'a mut String, f: F) -> i32
where
    F: for<'b> AsyncFn1<&'b mut String, Output = i32>,
{
    let v1: i32 = f(value).await;
    let v2: i32 = f(value).await;
    v1 + v2
}

async fn do_thing_all_values<'a, F>(values: &'a mut Vec<String>, f: F) -> i32
where
    F: for<'b> AsyncFn1<&'b mut String, Output = i32>,
{
    let mut total = 0;
    for v in values.iter_mut() {
        let result = do_thing_single_value(v, &f).await;
        total += result;
    }
    total
}
2 Likes
3

Thanks a lot for your answer! Very interesting this crate you suggested, I'm trying to understand how it works now!

But I guess what I don't understand is: why would we need HRTB in this case? Because it isn't a FnMut, so after the call to the function ends (or await is called in the async case), it shouldn't matter the lifetime of the argument we passed, should it? Is this a limitation of the borrow checker, or it's just impossible for it to reason it? Currently I'm unable to think of a scenario where this is really needed :thinking:

About the async version, you said:

You cannot specify the future type as a type parameter, because the future captures the mutable reference, so the future type depends on the lifetime.

Why we cannot do that? I actually tried to specify something like -> (Fut + 'b), don't recall exactly why it didn't work.

4

The concrete future type is determined by the implementation of the function, and is generated by the compiler. A type parameter is chosen by the caller, but in this case, it doesn't make sense to allow the caller to chose a type – it must be whatever the compiler emits during the desugaring of the async fn.

This is basically the exact same thing as the difference between RPIT and APIT that is mentioned numerous times on this forum. The future is an RPIT, but the equivalent of a generic parameter would be an APIT.

5

The future is an RPIT

This was very insightful! I'm rather new to Rust, so I didn't know much about this before, but reading about it here (Return position impl Trait in traits), I guess I understand it better now. I'll need to study this a bit more, but thanks a lot for clearing it up!

6

The more general concept you are looking for is "existential type".

7

You're describing a different problem. In this case, there are no should-be-RPITs-but-aren't in the original program — the problematic functions are accepting futures (produced by the function passed in), not producing them (except as ordinary async fns).

In general, a type that contains a lifetime is a different type if it contains a different lifetime. The reborrows have different lifetimes. So, going back to your original code, let's insert the explicit reborrows and give some names:

    let v1: i32 = f(&mut *value).await; // call this reborrow 'b1
    let v2: i32 = f(&mut *value).await; // call this reborrow 'b2

Let's suppose that the given function f returns an actually nameable future type, as if it were not an async fn result but but an explicitly implemented Future struct:

struct MyFuture<'t> {...}
fn thing_to_do<'t>(&'t mut String) -> MyFuture<'t> {...}

Then in the first line, the type of the result of f(&mut *value) is MyFuture<'b1> containing a &'b1 mut String, and in the second call it is MyFuture<'b2>. But MyFuture<'b1> and MyFuture<'b2> are different types, and the Fut type parameter can only be one of them. (Or, actually, it cannot be either of them, because 'b1 and 'b2 are lifetimes that start and end within the function call, and a type or lifetime parameter to the function cannot involve a lifetime that doesn't outlast the function call.)

The exact same logic applies to the unnameable async {} and async fn future types, even though you can never see the lifetime parameter.

2 Likes
8

I was referring to the fact that the desugaring of the return type of an async fn can't be a type parameter; maybe that's not what you meant by

but it certainly is what I thought, because the function signature above this assertion of yours is a plain async fn with no other functions (including f: F) accepting a future as an input – the only future in the signature is the return type of the async fn after desugaring:

9

Ah, I see. Let me expand. The type parameter I was referring to was Fut in the OP:

async fn do_thing_single_value<'a, 'b, F, Fut>(value: &'a mut String, f: F) -> i32
where
    F: Fn(&'b mut String) -> Fut,
    Fut: Future<Output = i32>,
    'a: 'b,
{

This signature is coherent considered by itself, but it doesn't allow the function body to compile, because it requires the &mut String passed to f to have lifetime 'b, but the body of the function calls f with shorter reborrows. The immediate next step to fix that is to switch to HRTB:

async fn do_thing_single_value<'a, F, Fut>(value: &'a mut String, f: F) -> i32
where
    F: for<'b> Fn(&'b mut String) -> Fut,
    Fut: Future<Output = i32>,
{

Now it is the case that using the parameter Fut is too restrictive because, being a parameter, it is a single typr, but we actually need to allow F's return type to capture the 'b HRTB lifetime. So in my original post I skipped over that intermediate state and went straight to removing both 'b and Fut from do_thing_single_value's generic parameters.

(The above signature would work for Fs which only used the &mut String immediately, and returned some impl Future that doesn't capture 'b — but async fns never do that. And, in general, the <F, Fut> pattern always works when F doesn't take any lifetime-bearing parameters.)

2 Likes
10

I did not know this, very insightful, thanks! I think with your explanation for this part I now mostly understand why we cannot specify a lifetime for this future. It seems async+lifetimes is something much more complex than I initially thought, I still have much to study and learn.

But apart from that, I still do not understand why these lifetimes matter in this case. After we await the future, aren't the references dropped with it? Shouldn't we be allowed to borrow again after that?

11

Nope, the function might have hung on to the reference. Proof that given the original signature, it is allowed to do so:

use std::future::Future;

async fn do_thing_single_value<'a, 'b, F, Fut>(value: &'a mut String, f: F) -> i32
where
    F: Fn(&'b mut String) -> Fut,
    Fut: Future<Output = i32>,
    'a: 'b,
{
    f(value).await
}

#[tokio::main]
async fn main() {
    let mut s = String::from("hello");

    {
        let stolen = std::cell::RefCell::new(Vec::new());
        do_thing_single_value(&mut s, |b| async {
            stolen.borrow_mut().push(b);
            0
        })
        .await;
        stolen.borrow_mut().last_mut().unwrap().push_str(" world");
    }

    println!("{s}");
}

This program prints hello world because the stolen reference was used to mutate s. If do_thing_single_value were allowed to call f twice, then stolen would contain two copies of the same mutable reference, which violates the borrowing rules.

This is a practical demonstration of the fact that when you write fn do_thing_single_value<'b>(...) you're making 'b a single lifetime that the caller gets to pick; here it's a lifetime that extends to where stolen is used afterward.

(None of this is specific to async; the exact same program could be written with ordinary Fns.)

1 Like
12

Well, now I just feel dumb. Maybe I forgot or was just confused, but I was thinking that a Fn wouldn't be able to mutate anything outside its body. Your example seems so obvious now, but that's probably underrating how clever it is!

Thanks a lot for all the explanations, I learned a lot today!