Capture the deref'ed value in closure

I think what we discovered here is that not only the compiler would infer how to capture the variables (and consequently the trait of the closure) - with or without the move keyword - to make the body of the lambda work, it even tries to apply "deref coercion" (at compile/desugar time) (it appears) as many times as it needs, so it can capture the right variable correctly.

borrow the borrow of the borrow

The compile doesn't just intelligently capture the variable by-value - best illustrated with this example where an additional move won't save us (it has to keep applying move/deref, until it compiles.

https://play.rust-lang.org/?version=stable&mode=debug...


struct ClosureWrapper<'a> {
    b: Box<dyn Fn() -> () +'a>
}

fn foo<'a>(v: &'a Vec<i64>) -> ClosureWrapper<'a> {
    let borrow_the_borrow = &v;
    let closure = || {borrow_the_borrow.len();};
    ClosureWrapper{b: Box::new(closure)}
}

fn main() {
    let v = vec![1,2,3];
    foo(&v);
}

Another weird part of this (undocumented?) behavior is that it compiles in the playground; but it doesn't on Compiler Explorer (https://godbolt.org/z/WjozhxGoW), or on my local machine. All tested on version 1.76.

I haven't looked at your code yet, but this is probably because you're not passing --edition=2021, and the difference is very likely RFC 2229.

2 Likes

Indeed activating the edition 2021 override (which I consider more convenient on Compiler Explorer than manually specifying the --edition argument) makes the linked code work.

And now looking at the code, the relevant difference between editions is that the updating capturing reborrows instead of taking references, and reborrow can outlast the reference it was reborrowed through.[1] It's a magical property of reborrows / references.

If you try with something that's not a reference, it fails, because the Deref trait forces the wrapper to remain borrowed / Wrap doesn't have the magical properties.


  1. It can outlast the liveness scope of the reference value; what the lifetime requirements are depends on what references are & or &mut. ↩︎

1 Like

Just like the improved closure capturing can capture individual fields if that’s all you use of a struct, e.g. a closure using foo.x (in read-only fashion; and after custom Deref::deref calls are desugared) will borrow &foo.x, not &foo.

Similarly thus, of only *foo is used (after custom Deref::deref calls are desugared) then &*foo would be captured instead of &foo. This only applies to types where * does not desugar to explicit calls to Deref::deref, so that’s &T, &mut T and Box<T>. For example Rc cannot do this:

use std::rc::Rc;

fn size_<T>(_: &T) {
    dbg!(std::mem::size_of::<T>());
}


fn main1() {
    let v = Box::new([1, 2, 3]) as Box<[i32]>;

    let f1 = || {
        let _ = v.len();
        // here, `v.len()` desugars to
        // `<[i32]>::len(&*v)`, and thus only `*v`
        // is accessed in this whole closure
    };

    size_(&f1); // ==> 16 bytes, from captured `&[i32]`

    let f2 = || {
        let _ = &v; // force whole v to be borrowed
        let _ = v.len();
    };

    size_(&f2); // ==> 8 bytes, from captured `&Box<[i32]>`
}


fn main2() {
    let v = &mut [1, 2, 3] as &mut [i32];

    let f1 = || {
        let _ = v.len();
        // here, `v.len()` desugars to
        // `<[i32]>::len(&*v)`, and thus only `*v`
        // is accessed in this whole closure
    };

    size_(&f1); // ==> 16 bytes, from captured `&[i32]`

    let f2 = || {
        let _ = &v; // force whole v to be borrowed
        let _ = v.len();
    };

    size_(&f2); // ==> 8 bytes, from captured `&&mut [i32]`
}


fn main3() {
    let v = Rc::new([1, 2, 3]) as Rc<[i32]>;

    let f1 = || {
        let _ = v.len();
        // here, `v.len()` desugars to
        // `<[i32]>::len(&*v)`, but in there `*v` uses custom
        // `Deref for Rc<T>`, further desugaring to
        // `<[i32]>::len(&*Deref::deref(&v))` in which
        // with `&v` the whole of `v` is borrowed!
    };

    size_(&f1); // ==> 8 bytes, from captured `&Rc<[i32]>`

    let f2 = || {
        let _ = &v; // force whole v to be borrowed
        let _ = v.len();
    };

    size_(&f2); // ==> 8 bytes, from captured `&Rc<[i32]>`
}

fn main() {
    main1();
    main2();
    main3();
}

Rust Playground

     Running `target/debug/playground`
[src/main.rs:4:5] std::mem::size_of::<T>() = 16
[src/main.rs:4:5] std::mem::size_of::<T>() = 8
[src/main.rs:4:5] std::mem::size_of::<T>() = 16
[src/main.rs:4:5] std::mem::size_of::<T>() = 8
[src/main.rs:4:5] std::mem::size_of::<T>() = 8
[src/main.rs:4:5] std::mem::size_of::<T>() = 8
1 Like

In the original code example thus, the “smarter” capturing does not apply any auto-deref during capturing.

Auto-deref happens before analysis of what’s captured, anyways:

borrow_the_borrow.len();

desugars to

<Vec<i64>>::len(&**borrow_the_borrow)

Note that this is fully desugared, as the * dereferences here are both for &T references, so these are built-in operations, not using Deref::deref.

Subsequent analysis for what needs to be captured then can deduce that for access to borrow_the_borrow, only **borrow_the_borrow is ever accessed (and in a read-only manner), so the closure captures include a shared immutable borrow of **borrow_the_borrow, which is a re-borrow &**borrow_the_borrow. This captured re-borrow then, for borrow-checking, is determined to be allowed to live for as long as the original v: &'a Vec<i64>, and no borrow-checking error occurs.


Note that this is my personal mental model of how the compiler handles these. I haven’t checked if this is actually true, or whether it does something else, like e.g. closure captures could be determined earlier and the logic re-implements some reasoning about the derefs and desugarings, or any other equivalent compiler implementation is possible.

1 Like

Here's some capture analysis on the OP between the editions:

And the difference:

2018

note: Min Capture borrow_the_borrow[] -> ImmBorrow
  --> src/main.rs:13:9
   |
13 | /         || {
14 | |             borrow_the_borrow.len();
   | |             ^^^^^^^^^^^^^^^^^ borrow_the_borrow[] captured as ImmBorrow here
15 | |         };
   | |_________^ borrow_the_borrow[] used here

2021

note: Min Capture borrow_the_borrow[Deref,Deref] -> ImmBorrow
  --> src/main.rs:14:13
   |
14 |             borrow_the_borrow.len();
   |             ^^^^^^^^^^^^^^^^^

And what the first lines mean is that in 2018, the closure captures &borrow_the_borrow where as in 2021, the closure captures &**borrow_the_borrow.

If you change the closure to be a move closure, they capture the same thing (borrow_the_borrow itself), which does not compile.

2 Likes

Thanks! #[rustc_capture_analysis] is awesome! Do you know if these internal attributes are documented anywhere? The best "list" I can find is rust/compiler/rustc_feature/src/builtin_attrs.rs at ed195328689e052b5270b25d0e410b491914fc71 · rust-lang/rust · GitHub.

If they're documented anywhere, I don't know where. I learned about that one from this comment (expand the collapsed part). Note that when it talks about closure captures, it's talking about 2018 captures until the "what about RFC 2229" section near the end (which is also where rustc_capture_analysis is mentioned/explained).

(If you keep reading the issue, it turns out they could get rid of the nested closure argument, using some relatively advanced lifetime bound gymnastics.)