Why is the lifetime of a mutable borrow via closure different when passing a parameter vs environment capture?

help
1

I don't quite understand why the following 2 closure variations behave differently:

#1 mutable borrow environment capture

    let mut vec = vec![1, 2, 3];
    let mut f = |x| { vec.push(x); };
    f(4);
    println!("{:?}", vec);
    f(5);
    println!("{:?}", vec);

#2 mutable borrow parameter

    let mut vec = vec![1, 2, 3];
    let f = |v: &mut Vec<usize>, x| { v.push(x); };
    f(&mut vec, 4);
    println!("{:?}", vec);
    f(&mut vec, 5);
    println!("{:?}", vec);

Since the closure does not return any reference, I initially expected the borrow of vec to behave the same way in both variations. However, while #2 works fine, #1 fails to compile with:

let mut f = |x| { vec.push(x); };
            ---   --- first borrow occurs due to use of `vec` in closure
            |
            mutable borrow occurs here
f(4);
println!("{:?}", vec);
                 ^^^ immutable borrow occurs here
f(5);
- mutable borrow later used here

I noticed that if I remove the call to f(5) then both examples work. I think I vaguely understand that this is related to the following explanation in the book: Closures: Anonymous Functions that Capture Their Environment - The Rust Programming Language ("Between the closure definition and the closure call, an immutable borrow to print isn’t allowed because no other borrows are allowed when there’s a mutable borrow.")

But I'm confused on these 2 points:

  • Why does the mutable borrow need to be "alive" before the closure is called?
  • If the compiler were to allow #1, is there any way to make analogous changes in both variations such that #2 is still valid while #1 has some invalid/undefined behavior? (If the answer is "no", I would conclude this falls into the "reject some correct programs but never accept incorrect programs" case.)
2

Closures capture variables from their environment at definition time, so the mut borrow of vec is stored in the closure's environment, and this borrow lasts for the lifetime of the closure variable. This capture occurs in example 1, but not in example 2 where the reference is passed as an argument instead. In example 1, the lifetime of the closure is until f(5) which overlaps with the println! and giving a simultaneous (im)mutable borrows error. If you comment out f(5), then the lifetime of the closure variable is shortened to f(4) which does not overlap.

On your second point, the two closures are interpreted differently by the compiler. Conceptually, the first is similar to having an anonymous struct which has a mutable reference to the vec "on construction", so this borrow cannot be "delayed until use" like you're implying. e.g. a closure like this

let mut vec = vec![1, 2, 3];
let mut f = |x| { vec.push(x); };

is roughly desugared to

struct Closure<'a> {
    // capturing the reference to the orig vec
    vec: &'a mut Vec<i32>,
}

impl<'a> Closure<'a> {
    // some helper method that implements the closure's body.
    fn call_mut(&mut self, x: i32) {
        self.vec.push(x);
    }
}

Importantly, the lifetime of the borrow of the Vec is the same as the lifetime of this anonymous closure struct ('a).

I think this article from Rusty Yato explains this idea of an "anonymous struct" more in detail: Closures: Magic Functions | Rusty Yato

2 Likes
3

or, if you remove the println!() in between the two invocations of the closure.

as pointed out, the environment is captured at the defsite of the closure, not at the callsite of it.

that's just the way it is implemented. if we were to allow the environment to be "(re-)captured" only at callsite, then it is impossible for the compiler to do type checking without global reasoning about the entire program.

extern "Rust" {
    fn foo(f: impl FnMut());
}

let mut vec = vec![1, 2, 3];
let mut f = |x| { vec.push(x); };
// impossible to know if this should typecheck or not
foo(f);
3 Likes
4

Thanks all for helping to clear this up. I was also continuing to think about this overnight while waiting for a reply and basically concluded the same thing, that it is by design that the lifetime of a borrow by capture begins when the closure is defined.

There appear to be many (in retrospect, obvious) reasons why this design choice makes more sense than "recapturing the environment at call site", one being that if the closure is passed around to a different scope then the environment may no longer have variables in scope (or alive) that need to be captured.

I think some of my confusion here came from holding on to a mental model from some other languages where basically everything is a pointer / managed for you at runtime. In that case, if you have some function-like where one argument is always the same, it's an easy simplification to put it directly into the implementation and reduce the number of parameters. I have learned that when trying to do this in Rust, I also need to consider the borrow checker carefully.

An aside on my 2nd question

For my 2nd question, I don't think it's well-formed because I was misunderstanding what it means when the compiler raises a borrow checker error. Such an error doesn't necessarily mean "if we hadn't caught this error, your program would have done something undefined/invalid/unexpected"; it just means some references have violated the principle of aliasing-xor-mutability. (newbie mistake)

Aliasing-xor-mutability is often explained in the context of avoiding data races, which makes it hard to understand why it matters in a single-threaded, simple context. I appreciated this blog post which explains the philosophy behind this foundational part of Rust's design: The Problem With Single-threaded Shared Mutability - In Pursuit of Laziness

5

actually, in all languages I know that support closures, the environment is captured when the closure is created, not when the closure is called, just like how rust does it.

what's unique to rust is the borrow checker, not how closures are implemented. languages with managed runtime typically solve memory problems with GC, while languages without GC usually just rely on conventions (thus NOT memory safe).

for example, if you translate your code rejected by rustc into its C++ equivalence, it compiles fine. (I have heard news that some people are experimenting with borrow checker for C++, but my C++ knowledge is very outdated, so I'm not considering it for this example)

// helper function to print values of `vector<int>`
extern void print_vector(std::vector<int> const &v);

int main() {
    auto vec = std::vector {1, 2, 3};
    auto f = [&vec](int x) { vec.push_back(x); };
    f(4);
    print_vector(vec);
    f(5);
    print_vector(vec);
}

granted, rust's borrow checker (or any other similar static analysis based checker) is not perfect (and can NEVER be "perfect". I use "perfection" in a more general sense, not specific to rust's definitions, i.e. it has neither "false positives" nor "false negatives" regarding invalid memory access). as you can see from this particular example, the C++ equivalent is safe, there's no memory corruption. actually, you can emulate the C++ behavior in rust using raw pointers, bypassing the borrow checker completely, you just need to be very careful not to create aliased references at the same time, which would be immediately UB.

let mut vec = vec![1, 2, 3];
let vec = &raw mut vec;
let f = |x| unsafe { (*vec).push(x); };
f(4);
unsafe {
    println!("{:?}", &*vec);
}
f(5);
unsafe {
    println!("{:?}", &*vec);
}

if you run this snippet through miri, no UB should be detected.

as an aside, here's a slight variant of your original example, which is very similar to the classic "introductory example to rust for C++ programmers". this example is inherently incorrect, in that if you translate it into C++ (or use raw pointers and unsafe in rust), it could crash the program at runtime.

let mut vec = vec![1, 2, 3];
let x = &vec[0];
let mut f = |x| { vec.push(x); }; // error[E0502]: cannot borrow `vec` as mutable because it is also borrowed as immutable
f(4);
println!("{:?}", x);
1 Like
6

While the first sentence here is true, the invariants of &raw mut and &mut are different. As you refer to at the end of the paragraph, aliasing a &mut _ is UB.

So the version with &mut _ fails not because the borrow checker is imperfect for this use case, but because it was preventing UB.

Some may say &mut should be less restrictive, like the closure-captured reborrow going "inactive" and "reactivated" in the example or such, but that would be a change to the language -- a part of the language that unsafe code authors can rely on. (As does the compiler, for optimizations, some of which will be lost if you do everything with &raw mut.)

1 Like