Lifetime of nested references?

I found this example on: cotigao .medium.com/mutable-reference-in-rust-995320366e22

Take a look at this code (won't compile): Rust Playground

fn main () {
    let mut i:i32 = 1;
    
    let j = {
      let x = &mut i; // let x =&i; // compile
      let y = &x;
      &**y
    };
}

The problem is if we change let x=&mut i to let x=&i, it will compile. The original post explains that because we need to "move" everything in the inner scope to the outside scope for j to work, but &mut can't be moved or copied hence the problem.

But then I experimented more: Rust Playground

fn main () {
    let mut i:i32 = 1;
    
    let j = {
      let x = &mut i; 
      &*x
    };
    // let k=&i; // won't compile
    j;
}

I found out by re-borrowing x, j basically extends the lifetime of x even x is a &mut and that works.

My questions:

  • why j couldn't extend lifetime of x in the first example where j=&**y?
  • let's assume re-borrowing could only extend the lifetime of the "direct" variable it's referring to, in the first example it's y, so basically j extends lifetime of y but not x that's why it doesn't compile, but then why change to let x=&i (immutable), it compiles?

I've been wrapping my head around these questions for a while... Appreciate anyone could help :pray:

Let's go over the cases:

fn main () {
    let mut i:i32 = 1;
    
    let j = {
      let x = &mut i;
      let y = &x;
      &**y
    };
}

First, it should be clear that it would not be possible for y to exist after x is destroyed, as y has a reference to x. But then when you type &**y you are performing the flatten operation from &'short &'long mut i32 to &'short i32. Since this gives j the same lifetime as y, it wont be possible for j to live longer than y can.

You cannot get an &'long i32 when flattening a mutable reference because any use of the mutable reference must have exclusive access to the thing it points at. To illustrate how it can go wrong:

fn incorrect_flatten<'short, 'long>(r: &'short &'long mut i32) -> &'long i32 {
    let ptr: *const i32 = &**r;
    // THIS UNSAFE CODE IS INCORRECT
    unsafe {
        &*ptr
    }
}

fn main () {
    let mut i:i32 = 1;
    
    let x = &mut i;
    let y = &x;
    let j: &i32 = incorrect_flatten(y);
    *x = 2;
    println!("{}", j);
}

This will compile and print 2, but the compiler guarantees that the value that an immutable points at doesn't change between any two uses of that immutable reference, which is violated in the above since the value that j points at changes from 1 to 2. If we were dealing with a Vec<i32> or similar, it would be easy to wreck havoc using the above.


fn main () {
    let mut i:i32 = 1;
    
    let j = {
      let x = &i;
      let y = &x;
      &**y
    };
}

This compiles because with immutable references, the flatten operation is actually &'short &'long i32 to &'long i32. Basically all that's happening here is that we are copying the value that y points at, and y points at a value of type &'long i32, so that's the type we get.


fn main () {
    let mut i:i32 = 1;
    
    let j = {
      let x = &mut i; 
      &*x
    };
    // let k=&i; // won't compile
    j;
}

Here there is nothing that actually ties the lifetime on j to the value x, so there's no problem. The mutable borrow you perform when you create x is simply extended until the last use of anything derived from x, so i remains mutably borrowed until the last line.

It doesn't compile when you add in k because i is still mutably borrowed at that point. Downgrading a reference doesn't downgrade the kind of borrow it was created with.

5 Likes

First, some general notes.

The article is somewhat dated, as some of the examples that errorred at the time compile under NLL (non-lexical lifetimes).

You can definitely move a &mut, but you can't copy it.


Back to your examples. It may be instructive to see other ways in which they can fail, and also how they can succeed.

Let's play with this one a bit more. The original error (today) is:

error[E0597]: `x` does not live long enough

x is dropped at the end of the block, but a borrow to x is considered stored in j.

How about this variation?

    let mut i:i32 = 1;
    let j = {
      let x = &mut i;
      let y = &x;
      &* { *y }
      // ^    ^ new
    };

The error is now:

error[E0507]: cannot move out of `*y` which is behind a shared reference

The braces forced a move, and you can't move from behind a shared reference. It can't be a copy, because *y is a &mut i32, and those don't implement Copy. But shared references do implement Copy, and the example does (continue to) work with those.

Note that this means: the borrow stored in x may have a lifetime longer than the lifetime of x itself! If this is surprising at first, consider this situation:

fn foo() {
    // `s` doesn't last longer than `foo` but contains a static borrow
    let s: &'static str = "bar";
    // This fails because `s` doesn't live long enough
    let ss: &'static &str = &s;
}

How about this version then?

    let mut i:i32 = 1;
    let j = {
      let x = &mut i;
      let y = &x;
      &{ **y }
    };

It compiles. Why? Now we're moving out an i32, which implements Copy -- and then we're taking a reference to this temporary. And the temporary can live longer than the block. However, if you try this with a String instead of an i32, you'll get the "can't move from behind a shared reference" error again (but for **y this time), because String doesn't implement Copy.


Alice did a better job explaining why your example that works compiles; the only thing I would highlight is the "variables can hold references longer than themselves" observation from above. That's why the borrow created for x can live longer than x itself. In particular, the example is not relying on copying (i.e. it works with String as well as i32).


Can we make x in the original example to live long enough some other way (including with non-Copy types like String)? Yes, we can:

    let mut i: String = "".into();
    let x; // Allow lifetime greater than the block below
    let j = {
      x = &mut i;
      let y = &x;
      &**y
    };
    j;
1 Like

Thank you @alice for responding :blush:, really appreciated!
I'm still trying to understand your answer:

// won't compile                  |         compile
    let j = {                     |         let j = {         
      let x = &mut i;             |           let x = &mut i; 
      let y = &x;                 |           &*x;
      &**y                        |
    };                            |         };

On the left: &'short &'long mut i32 -> &'long i32 - impossible
On the right: &'long mut i32 -> &'long i32 - possible, is this what you meant ? Because you also mentioned any use of the mutable reference must have exclusive access to the thing it points at so I'm quite confused, I thought &'long mut i32 would block any shared reference being created?

And could you elaborate a bit more on this one also?

Here there is nothing that actually ties the lifetime on j to the value x , so there's no problem.

And in the 2nd example you mentioned:

This compiles because with immutable references, the flatten operation is actually &'short &'long i32 to &'long i32 . Basically all that's happening here is that we are copying the value that y points at, and y points at a value of type &'long i32 , so that's the type we get.

So the compiler can infer from the context and chooses to copy the value? Could you point me to somewhere in the doc has more info on this?

Thank you :smiley:

Thank you @quinedot for your detailed answer as well :innocent:, this indeed helps me understanding more on other situations that I might face with, especially those on move notation:

It may be instructive to see other ways in which they can fail, and also how they can succeed.

This applies to shared reference being created independently of the &'long mut i32. I.e. if there's a variable x: i32, and you create let y = &mut x;, then you cannot also create let z = &x while the reference in y is still alive.

On the other hand re-borrowing the &'long mut x as a &'long x is possible, even multiple times

fn f<'long>(x: &'long mut i32) {
    let y: &'long i32 = &*x;
    let z: &'long i32 = &*y;
    println!("{}", y);
    println!("{}", z);
}

furthermore, reborrowing a mutable reference can often happen implicitly

fn g<'long>(x: &'long mut i32) {
    let y: &'long i32 = x;
    let z: &'long i32 = y;
    println!("{}", y);
    println!("{}", z);
}

(playground)


The main problem with your original example

fn main () {
    let mut i:i32 = 1;
    
    let j = {
      let x = &mut i;
      let y = &x;
      &**y
    };
}

is that the variable x only exists inside the block, hence the reference to x cannot be alive after that block. So there's two parts to the problem:

  • A reference to x is created, i.e. the &x assigned to y
  • This reference is used in a way that requires that reference to be alive for longer as the block in which x is in scope.
    • This is due to &**y which turns y: &'short &'long mut i32 into &'short i32, hence producing a reference whose lifetime is bound to the lifetime of that y = &x reference.

Now your other two settings both drop one of these parts:

  • With

    fn main () {
        let mut i:i32 = 1;
      
        let j = {
          let x = &i;
          let y = &x;
          &**y
        };
    }
    

    the second part of the problem is gone: Now &**y turns y: &'short &'long i32 into &'long i32. The lifetime of this reference is no longer connected to how long x can be borrowed (from the expression &x) but instead only connected to how long i can be borrowed (from the expression &i). IMPORTANT: Even though &i is assigned to a short-lived variable x, it can still contain a reference that lives longer than the scope of the variable it's assigned to! References can be moved around - shared references also copied - it doesn't really matter what kind of variable they are initially assigned to; the transformation &'short &'long i32 -> &'long i32 does simply copy the reference out of y; it's basically a special case of dereferencing &'short T -> T where T: Copy. The fact that you're writing &**y instead of simply *y is making this a bit less intuitive, but the compiler somehow understands that there's really no problem, i.e. that it's essentially the same as *y.

  • With

    fn main () {
        let mut i:i32 = 1;
      
        let j = {
          let x = &mut i; 
          &*x
        };
        // let k=&i; // won't compile
        j;
    }
    

    instead, the first part of the problem is gone. Here, we don't create any reference to a local varaible inside of the block in the first place. There's no "&x" anywhere.


Edit:

In the last example above, it may be confusing that you're re-borrowing from a reference contained in x for longer than the variable x exists; but really for soundness all that matters for the re-borrowing is the lifetime argument of the type of the reference value that x contains, not how long the containing variable exists.

Now, there's a problem: When x goes out of scope, it gets dropped; dropping a value do something, so in general it would need access to what's inside. You cannot, for example, reborrow a reference inside of a Vec for longer than that Vec lives.

fn foo() {
    let x = &mut 0_i32;
    let z;
    {
        let mut y = vec![x];
        z = &mut *y[0];
        // y dropped
    }
    z; // doesn't compile
}

But for types that don't implement Drop, like &mut T, it could work. So e.g. for for fields in structs not implementing Drop

struct S<T>(T);

fn foo() {
    let x = &mut 0_i32;
    let z;
    {
        let y = S(x);
        z = &mut *y.0;
        // y dropped
    }
    z; // compiles fine
}

I don't have a source on that, but it seems to me that re-borrowing a mutable reference (almost) always works if moving that reference would also have worked. (This means it also works with Box with has some special - somewhat magical - properties so you can move out of field of a struct behind a Box. So this works (link).) This also interacts nicely with the fact that Rust will quite often introduce implicit re-borrows of mutable references everywhere it might make sense; this would be a bad feature if it were able to make code no longer compile when the same code when it would move the reference would still compile.

E.g. on &mut T method arguments, you'll have implicit re-borrows, as demonstrated by

fn foo() {
    let x = &mut 0_i32;
    bar(x); // x NOT moved
    bar(x); // cann call again
}

fn bar(_: &mut i32) {}

This means the above is essentially the same as

fn foo() {
    let x = &mut 0_i32;
    bar(&mut *x);
    bar(&mut *x);
}

fn bar(_: &mut i32) {}

But if you would do something like

fn foo() {
    let mut i = 0;
    let y;
    {
        let x = &mut i;
        bar(x);
        y = bar(x);
    }
    println!("{}", y);
}

fn bar(x: &mut i32) -> &mut i32 { x }

then the second call, y = bar(x) is expected to move the&mut i reference out of that block; the fact that it's equivalent to y = bar(&mut *x) must not make the compilation fail.

2 Likes

wow, thank you so much @steffahn, love the detailed explanation, that's awesome :exploding_head: It exceeds my expectation on not just answering the question but also offering more insights. Let me have some time to grind the answer and circle back.

1 Like

Yeah, we have &'short &'long mut i32 -> &'long i32 impossible but &'long mut i32 -> &'long i32 possible.

Regarding the exclusive access stuff, it is possible to use your mutable reference to create a sub-reference. What exclusive means here is that you cannot use the mutable reference again until after you are done using the sub-reference.

The reason you can't flatten here is that the compiler loses the connection between the mutable reference and your sub-reference if you could flatten and get a long lifetime, but it needs that connection to enforce that no uses of x happen before the last use of the sub-reference.

When you perform a reborrow on a mutable reference x, yielding a sub-reference j, all the compiler requires is that the lifetime annotated on j is a subset of the lifetime annotated on x, and that there are no uses of x within the lifetime annotated on j.

If you never use x again once you've created j, then the second requirement is trivially satisfied, and only the second one is relevant.

It always copies when the inner reference is immutable. We only really need to talk about reborrows and sub-references when we are dealing with mutable references.

1 Like

Love you all guys for awesome responses :metal:, they offer a lot of details and insights into this topic and I've learnt a lot of new things thru-out these days! (I also realized how the official book and Rustonomicon need a lot more updates to incorporate these information actually, I din't find answers in there or maybe because of my ignorance :crying_cat_face:).

The thing that did click into my mind and help unblocking my thought process is this paragraph:

The main problem with your original example

fn main () {
    let mut i:i32 = 1;
    
    let j = {
      let x = &mut i;
      let y = &x;
      &**y
    };
}
is that the variable x only exists inside the block, hence the reference to x cannot be alive after that block.

So I consider @steffahn 's answer is the solution! It enabled me to understand the other answers of @quinedot and @alice (sorry I'm so noob :pray:).

Deep dived into this topic, I found another thread in which @steffahn explained in very detailed way about re-borrowing, I would consider it to be "The unofficial guide on re-borrowing", and encourage fellow users to read thru it. Lifetime and borrow are hard to fully acquire so it's definitely worth to invest your time guys. In that thread, @quinedot also provided other 2 articles which are super helpful and informative:

I also found a similar question on this thread, and I hope the answers here could help you as well @zylthinking .

Last but not least, I was hopeless because I couldn't find any answer to my question before, and my last option was signing up on this user page and immediately creating this thread. This community has amazed me :exploding_head:, @alice responded in less than 1 hour, followed by @quinedot @steffahn a few hours after that. You guys did not only answer the question with awesome details and demo code (I couldn't think of anyone on the internet could spend that much of time to try my examples, not even count to create a demo to help me to understand :pray:), but also replied to my follow-up questions as well as provided further resources for me to learn more.
This is phenomenal I would say. So I would like to thank you all and encourage you guys to continue keeping this up like what you guys have been doing so far :clap:, this community needs people like you! :heart:

(I'll go grinding your other answers on other topics, so don't be surprised if you receive :heart: sometimes :smiley: ).

1 Like

Those books are not complete descriptions of the language, but introductory material. For maximally in-depth information always also consider looking into The Rust Reference; some information is also only in the standard library docs; some information is poorly documented, e.g. only in some RFCs or no-where at all. Trying out small examples and "talking with" the compiler (i.e. seeing what compiles and what doesn't) is also a good way to find out more abound weird special/corner cases. Last but not least, information from blog posts, looking into (the source code) of the standard library or popular crates, as well as answers in this very forum complete the picture of where to learn from, at least for me.

I can't exactly remember where I learned which part of what I told you in this thread; the final coherent picture / presentation of course also is in some ways just my own personal interpretation of or intuition about the rules of the language. Undoubtedly, there are things that should belong into the The Rust Reference but aren't there yet.

2 Likes

The code cant be compiled


fn foo() {
    let x = &mut 0_i32;
    let z;
    {
        let y = S(x);
        z = &mut *y.0;
        // y dropped
        drop(y);
    }
    z; // compiles failed
}

Right... an explicit call to drop is sometimes impossible even though logically (or even actually) the compiler drops the variable in that same place anyway. The reason is that explicit call to drop will always require access to the variable (either unique access, and moving out of the variable when it's not [known to be] 'Copy' or at least shared access when it is 'Copy') as far as the borrow checker is concerned, even though the type of the value being dropped might not even do anything at all upon being dropped.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.