Copy vs Reborrow - what happens from the point of view of lifetimes when accessing reference fields through &mut self?

help
1

EDIT: see below for a summary of my understanding trying to put all the great replies in a single place

Here's the reduced code showing what I'm trying to understand:

struct RefHolder<'a> {
    data_mut: &'a mut [u8],
    data_shared: &'a [u8],
}


impl<'a> RefHolder<'a> {

    //1) yield a shared reference to a subslice with the lifetime of the holding struct 
    fn yield_wide<'fun>(&'fun mut self, pos: usize) -> &'a [u8] {
        &self.data_shared[pos..]
    }

    //2) yield a shared reference to a subslice with the (narrower) lifetime of the function scope
    fn yield_narrow<'fun>(&'fun mut self, pos: usize) -> &'fun [u8] {
        &self.data_shared[pos..]
    }

    // 3) yield an exclusive reference to a subslice with the lifetime of the holding struct
    // NOTE: need to constrain 'fun: 'a as per compiler suggestion
    fn yield_mut_wide<'fun: 'a>(&'fun mut self, pos: usize) -> &'a mut [u8] {
        &mut self.data[pos..]
    }
   
    //4) yield an exclusive reference to a subslice with the (narrower) lifetime of the function scope
    fn yield_mut_narrow<'fun>(&'fun mut self, pos: usize) -> &'fun mut [u8] {
        &mut self.data[pos..]
    }
}

I got into this while trying to better understand what GATs bring to the table, which drove me down the rabbit hole of trying to implement a mutable iterator which finally got me to a case similar to the one shown above.
Here are my questions:

  1. First I'd like to check if the following reasoning is correct for why I need to put the 'fun: 'a bound in case 3: my understanding is that the signature of the function is promising to return something tied to the lifetime of the RefHolder instance, but since non-'static returned references must originate from (reference) function arguments, in this case the returned value will be tied to the lifetime of &mut self which the compiler has to pessimistically assume being shorter than 'a and hence it asks for a guarantee that 'fun is at least as long as 'a

  2. Regardless, why the same bound is not required for the corresponding (but non-mutable) case number 1? Is the compiler performing some kind of implicit lifetime augmentation because of non-mutability? Has it anything to do with variance/subtyping kicking in for the non-mutable case? I'm quite confused here...

Thanks,
Andrea

2

Immutable references are Copy, so you don't have to re-borrow them, you can simply copy a longer-living (shared) reference from behind self.

2 Likes
3

Yes, for shared loans there are implicit shenanigans happening. Rust calls it variance.

1 Like
4

sub-typing can be thought of the answer to the question: "is this assignment valid?". e.g. given left hand side is type A, and right hand side is type B, an assignment is valid if B is subtype of A. that's it. by assignment, I mean conceptually, not syntactically, e.g. the return type of a function and the actual value that is returned can be thought of as left hand side and right hand side of an assignment, respectively.

the difference you observed is not because lifetimes in shared and exclusive references are treated differently, but simply the fact that &'a T is Copy, while &'a mut T can only be moved. if you break down the steps of the evaluation of the expression and spell out their type explicitly, it would be much clearer:

for yield_wide, the return type is &'a [u8], we can break down the one liner expression &self.data_shared[pos..] into steps:

 // conceptually, the expression is evaluated in this order:
 // step 1: self: &'fun Self, note `mut` is irrelevant in this case
 // step 2: self.data_shared: &'a [u8]
 // step 3: self.data_shared[pos..]: [u8]
 // step 4: &self.data_shared[pos..] &'_ [u8]
fn yield_wide<'fun>(self: &'fun Self, pos: usize) -> &'a [u8] {
    // because shared reference is Copy, if we spell out the `Clone` type, you will find the `'fun` lifetime is irrelevant!
    // fn clone<'fun>(&'fun self) -> Self;
    let data_shared: &'a [u8] = Clone::clone(self.data_shared);
    // the index signature give the same 'a lifetime
    return core::ops::Index::index(data_shared, pos..)
    // next lines is conceptual when use sqaure bracket syntax, not really happening in reality
    //let slice: [u8] = data_shared[pos..];
    //let slice: &'_ [u8] = &slice;
    //return slice;
}

two things to note:

  • in step 2, we get a &'a [u8] regardless of the 'fun lifetime, this is the key point: shared reference is Copy !
  • in step 3, conceptually you get this unsized [u8] because the squaure bracket index syntax is desugared to the derefenced type. but you cannot bind this unsized intermediate value to a variable, you must re-borrow it immediately (and the optimizer knows this, so the step actually didn't happen in reality). you can skip the sugar syntax but call the core::ops::Index::index() trait explictly, its excatly equivalent.

if you get the idea, you'll find out that if you do the similar break down for yield_mut_wide(), you can't actually get a copy of &'a mut [u8], you must borrow if from self, that's why you need the 'fun: 'a bounds.

also notice, it's not the function signature that requires the bounds, it's the implementation. for instance, if you change the yield_wide() to return a slice of data_mut instead of data_shared, you will get an lifetime error too:

    fn yield_wide<'fun>(&'fun mut self, pos: usize) -> &'a [u8] {
        &self.data_mut[pos..]
    }
2 Likes
5

Wow! Thanks for the detailed reply.
I think I see what you're saying, and I probably found a flaw in my mental model, can you please check if I'm looking at this from the right angle now?

In general

struct S<'a>
{
   field: &'a u8
}
impl<'a> S<'a>
{
  fn some_method<'fun>(&'fun self, ....) -> whatever
  {
     let access_to_field_through_self: (inferred &'????) = self.field;
  }
}

I thought that the variable access_to_field_through_self would have always been inferred to be tied to the 'fun lifetime

..
let access_to_field_through_self: (inferred &'fun u8) = self.field
...

(i.e. the lifetime of the local borrow of self) since the access to field was "going through" self.
That's why I was thinking that the compiler was performing some kind of "lifetime increase/augmentation" when accepting the yield_wide function, because I thought we would always "start from a shorter lifetime".
In other words, I thought that any expression going through self would be tied to (the possibly shorter) self's lifetime, no matter what and in some cases (when no mutability was involved) the compiler was able to "stretch" it.

But from your reply I think I should see it in this other way:
expressions accessing fields through self always infer the (possibly longer) lifetime of the value they are accessing, i.e.

..
let access_to_field_through_self: (inferred &'a u8) = self.field
...

but when returning from a function, that (possibly longer) lifetime is preserved only when the reference is Copy.
In the case of exclusive references, the compiler uses re-borrowing when returning the value from the function and that cause its lifetime to become the shorter local 'fun

Does this make some kind of sense?

6

This is and must be the case for &mut references to maintain their exclusivity. But an & reference is simply copied like any other value may be copied, and the copy has the same type: self.field is of type &'a u8, so access_to_field_through_self is also of type &'a u8.

3 Likes
7

no, it's incorrect. the compiler tries to use the "apparent" type first, only when the apparent type doesn't work will it try coersion rules. but multi level deref coersion would result to the "shorter" lifetime, not the "longer" lifetime.

in the case of shared reference, it's not because the compiler coerced to the "longer" lifetime, it's because the type of the field itself (i.e. shared reference type) is Copy.

let's ignore lifetimes for a moment, and just take a look at the simplest example:

struct Foo<T> {
    pub x: T,
}
impl<T> Foo<T> {
    // this will not work: error: cannot move out of `self.x`
    fn get_x_1(&self) -> T {
        let x: T = self.x;
        x
    }
    // this will work
    fn get_x_2(&self) -> T where T: Copy {
        let x: T = self.x;
        x
    }
    // this will also work
    fn get_x_3(&self) -> T where T: Clone {
        let x: T = self.x.clone();
        x
    }    
}

what's point? well, if you change it a little bit:

type NonRef = usize;
type SharedRef<'a, T> = &'a T;
type ExclusiveRef<'a, T> = &'a mut T;

let foo1: Foo<NonRef> = todo!();
let foo2: Foo<SharedRef<'a, usize>> = todo!();
let foo3: Foo<ExclusiveRef<'a, usize>> = todo!();

// works, no lifetime whatsoever, just an `usize`
let x: usize = foo1.get_x();

// works, although the alias `SharedRef` contains a life time parameter,
// it is just a type, a shared reference type satisfies the `Copy` bounds
// it's **NOT** the type being referenced, but the **reference** itself is `Copy`!
let x: SharedRef<'_, usize> = foo2.get_x();

// doesn't work, IMPORTANT: exclusive reference is not even clonable! 
let x: ExclusiveRef<'_, usize> = foo3.get_x();

now let's see another example modified from your original code:

struct Foo<'a> {
    x: &'a mut usize
}

impl<'a> Foo<'a> {
    /// this works, intuitively, automatically coersion to the `'fun` lifetime
    fn get_x_1<'fun>(self: &'fun Self) -> &'fun usize {
        //&self.x
        self.x
        // the extra & operator is unnecessary, but the deref coersion rule is very confusing
        /*
        return self.x;              // compiles
        let x = self.x; return x;   // error
        let x = &self.x; return *x; // compiles;
        let x = &*self.x; return x; // compiles
        */
    }

    /// this requires the sub typing bounds
    fn get_x_2<'fun>(self: &'fun Self) -> &'a usize where 'fun: 'a {
        self.x
        // think of it in two steps: the sub typing bounds make the second assignment valid,
        /*
        let x: &'fun usize = self.get_x_1();
        let retval: &'a usize = x;  // this need 'fun to be sub type of 'a
        return retval;
        */
    }
}
2 Likes
8

It's not really about the compiler being pessimistic. The function declaration declares an API that your function body and callers must both adhere to, in order to ensure soundness (and reasonable program behavior).

Without the bound, the rule that prevents the API from being met is that you can't get a &'long mut T by going through a &'short mut &'long mut T. That is, without the bound, you were attempting to return a &'fun mut [u8].

Incidentally here

fn yield_mut_wide<'fun>(&'fun mut self, pos: usize) -> &'a mut [u8] {

self is a &'fun mut RefHolder<'a>, and that nesting means there's an implicit bound already: 'a: 'fun. When you add another bound 'fun: 'a like the compiler suggests, the two bounds together imply that 'a and 'fun are the same, and you effectively get

fn yield_mut_wide(&'a mut self, pos: usize) -> &'a mut [u8] {

This &'a mut Thing<'a> pattern is an anti-pattern as it means Thing<'a> is exclusively borrowed forever, and cannot be used ever again (except via the &'a mut).

The aforementioned "can't get a &'long mut T by going through a &'short mut &'long mut T" situation comes up with some types of mutable iterators, and can sometimes require using raw pointers and unsafe depending on the data structure (e.g. when you're not just wrapping some other iterator).

But there are sometimes safe ways to deal with it. For slices in particular, you can move an inner &'long mut [U] out of the outer &'short mut:

struct MyIterMut<'a, U> {
    slice: &'a mut [U],
}

impl<'a, U> Iterator for MyIterMut<'a, U> {
    type Item = &'a mut U;
    fn next(&mut self) -> Option<Self::Item> {
        let slice = std::mem::take(&mut self.slice);
        let (first, rest) = slice.split_first_mut()?;
        self.slice = rest;
        Some(first)
    }
}

Here, the std::mem::take returns the inner &'a mut[U] (which is possible because &mut [V] implements Default for any V and any lifetime by returning an empty slice). That allows you to call split_first_mut with the lifetime you require for the return value. (Then you put the rest of the slice back into *self.)

When you are reborrowing through references, you can start at the thing you're reborrowing and work your way outward; the lifetime of the reborrow will be limited to the lifetime of the first (inner-most) shared reference if there is one, or the last (outer-most) exclusive reference otherwise.

These are the "reborrow constraints" discussed here.

6 Likes
9

... and on that way you explicitly express the invariance of &'fun mut RefHolder<'a> over RefHolder<'a>, correct? :slight_smile:

10

I'm not sure what you mean, but 'a being invariant in &'_ mut Thing<'a> is part of the reason why it's such an anti-pattern. This works because 'x is covariant &'_ S<'x> , so &'a S<'b> coerces to &'a S<'a> and you don't have to borrow the struct for the rest of its validity.

11

Yes, it's an anti-pattern. I just wanted to mention that explicitly expressing the bound 'fun: 'a is necessary due to the invariance of &mut T over T according to the Nomicon.

12

No, it's because you can't reborrow &'a mut through a &'fut mut &'a mut, so instead you need a &'a mut &'a mut.

You can't borrow a &'a through a &'fut &'a mut either, so this gives you the same error even though &T is covariant over T.

struct RefHolder<'a> {
    data: &'a mut [u8],
}

impl<'a> RefHolder<'a> {
    fn yield_mut_wide<'fun>(&'fun self, pos: usize) -> &'a [u8] {
        &self.data[pos..]
    }
}
2 Likes
13

Oops! You're right of course. My bad, sorry for the confusion.^^

14

I think this is an important point that escapes me, could you please elaborate a bit what you mean?
IIUC, you're saying that for &mut references the inferred lifetime when accessing them through self is the one of the local borrow of self (i.e. 'fun), but I don't understand what is the implication regarding "to maintain their exclusivity".

Thanks.

15

If you could obtain a &'long mut T from behind a &'short mut T, then the lifetime of the returned reference would not be tied to the originating reference (pretty much by definition), and so you could create two parallel mutable references.

fn extend_mut_lifetime<'short, 'long: 'short, T>(ptr: &'short mut T) -> &'long mut T {
    &mut *ptr // unsound reborrow
}

let mut x = 42;
let long_ref_1 = extend_mut_lifetime(&mut x);
let long_ref_2 = extend_mut_lifetime(&mut x);
// BOOM, UB: simultaneous mutable references to `x`

Here's the same example forced to compile with unsafe, which Miri marks to be UB, as expected.

2 Likes
16

For nested borrows, this is Example 3 in the reborrowing constraints linked before.

Example 3. The previous example showed how a borrow of a shared reference can expire once it has been dereferenced. With mutable references, however, this is not safe. Consider the following example:

 let foo = Foo { ... };
 let p: &'p mut Foo = &mut foo;
 let q: &'q mut &'p mut Foo = &mut p;
 let r: &'r mut Foo = &mut **q;
 use(*p); // <-- This line should result in an ERROR
 use(r);

The key point here is that we create a reference r by reborrowing **q; r is then later used in the final line of the program. This use of r must extend the lifetime of the borrows used to create both p and q. Otherwise, one could access (and mutate) the same memory through both *r and *p. (In fact, the real rustc did in its early days have a soundness bug much like this one.)

Because dereferencing a mutable reference does not stop the supporting prefixes from being enumerated, the supporting prefixes of **q are **q, *q, and q. Therefore, we add two reborrow constraints: 'q: 'r and 'p: 'r, and hence both borrows are indeed considered in scope at the line in question.

As an alternate way of looking at the previous example, consider it like this. To create the mutable reference p, we get a "lock" on foo (that lasts so long as p is in use). We then take a lock on the mutable reference p to create q; this lock must last for as long as q is in use. When we create r by borrowing **q, that is the last direct use of q -- so you might think we can release the lock on p, since q is no longer in (direct) use. However, that would be unsound, since then r and *p could both be used to access the same memory. The key is to recognize that r represents an indirect use of q (and q in turn is an indirect use of p), and hence so long as r is in use, p and q must also be considered "in use" (and hence their "locks" still enforced).

1 Like
17

Ok, I think I might finally have a much clearer idea now.

I edited the original title of this post as I think it now better expresses what was the root of my confusion.

I tried to squeeze out a summary from all the great answers I got here, hoping it doesn't contain anything obviously wrong and that it might be of some help to someone else.

Here it goes:

Consider a struct containing some reference fields:

struct RefHolder<'a> {
    data_exclusive: &'a mut [u8],

    data_shared: &'a [u8],
}

And then a function that takes an argument of type &mut RefHolder:

fn some_function<'short, 'long>(struct_ref: &'short mut RefHolder<'long>) {…}

I realised that, starting from the original question, what really got me stuck was understanding
what happens from the point of view of lifetimes when accessing reference fields of a structure through struct_ref?

Note that this is a generalization of the more specific case of a method of such RefHolder struct:

impl<‘long> RefHolder<‘long>
{
…
    fn some_method<‘short>(&’short mut self) {…}
}

but the question remains the same: what happens from the point of view of lifetimes when accessing reference fields through &mut self?

The fundamental points, as suggested by quinedot, kpreid, H2CO3 are that

  • A) an & reference is Copy and so it is simply copied, and the copy has the same type
  • B) you can't get a &'long mut T by going through a &'short mut &'long mut T
  • C) If you could obtain a &'long mut T from behind a &'short mut T, then the lifetime of the returned reference would not be tied to the originating reference (pretty much by definition), and so you could create two parallel mutable references.

So when you access a structure’s reference field

  1. If the fields is a shared reference, it can get copied “carrying its lifetime along with it” (‘long in this case, i.e. the one annotated in the containing struct), possibly coercing it to a ‘short-er one due to subtyping / covariance (since &’short [mut] RefHolder<‘long> implies ‘long: ‘short)

  2. If the reference is exclusive, then it not Copy and not Clone and the options are
    2.A) You can move it and it would carry along its lifetime. But you have to keep in mind that in this case you can’t simply move since it is behind &self. When possible, using something like std::mem::take, std::mem::replace, std::mem::swap can effectively make you able to do a "move" and get hold of the original ‘long lifetime

    2.B) You reborrow it, which you can do explicitly (via &mut *) or implicitly assigning to an explicitly typed variable or e.g. passing-to / returning-from a function. When reborrowing, points B & C above kick in and “to maintain exclusivity” you can only end up with a ‘short lifetime

See the following example code as a summary:

struct RefHolder<'a> {
    data_exclusive: &'a mut [u8],

    data_shared: &'a [u8],
}

impl<'long> RefHolder<'long> {

    fn some_method<'short>(&'short mut self) {

    /**** OK ****/
    // Shared
    let _ok: &'long [u8] = self.data_shared; // & are Copy, original ‘long preserved
    let _ok: &'short [u8] = self.data_shared; // same, but can also be covariantly coerced to ‘short

    // Exclusive
    let _ok: &'short mut [u8] = self.data_exclusive; // reborrow to ‘short
    // The following is OK too, but can't coexist with the previous one
    // let _ok: &'long mut [u8] = std::mem::take(&mut self.data_exclusive); // use std::mem::take to “move”, original ‘long preserved

    /**** NOT OK ****/
    // (Only) Exclusive
    //let _ko = self.data_exclusive; //Attempt to move. ERROR: cannot move out of `self.data_exclusive` which is behind a mutable reference
    //let _ko: &'long mut [u8] = self.data_exclusive; //Attempt to get a &'long mut T by going through a &'short mut &'long mut T ERROR: type annotation requires that `'short` must outlive `'long`

}

}
18

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.