Why does changing from &T to &mut T break my lifetimes?

I've recently published a crate (https://crates.io/crates/rsor), which mostly works as intended, but there is one lifetime issue I need help with.

Let me start with an example that does work correctly (AFAICT):

The main struct in my crate has a method like this:

pub fn fill<'a, 'b, F>(&'a mut self, f: F) -> &'a [&'b T]
where
    F: FnOnce(Vec<&'b T>) -> Vec<&'b T>,
{
    /* ... */
}

The idea behind the different lifetimes is that lifetime 'a depends on my struct, but lifetime 'b only depends on the input to the method, so it should be able to live on even after my struct is destroyed.

And alas, this seems to work in a (quite contrived) example:

use rsor::Slice;

fn main() {
    let data = 'a';
    let outer_reference = {
        let mut reusable_slice = Slice::new();
        let chars = reusable_slice.fill(|mut v| {
            v.push(&data);
            v
        });
        chars[0]
    };
    assert_eq!(*outer_reference, 'a');
}

My struct reusable_slice goes out of scope, but outer_reference still lives.

Now to my problem: I would like to do the same thing with mutable references.

I have a mutable version of the method mentioned above:

pub fn fill_mut<'a, 'b, F>(&'a mut self, f: F) -> &'a mut [&'b mut T]
where
    F: FnOnce(Vec<&'b mut T>) -> Vec<&'b mut T>,
{
    /* ... */
}

This should be exactly the same thing as before, except the immutable references have been replaced with mutable references.

I would like to have a similar example as above, just with mutable references:

use rsor::Slice;

fn main() {
    let mut data = 'a';
    let outer_reference: &mut char = {
        let mut reusable_slice = Slice::new();
        let chars = reusable_slice.fill_mut(|mut v| {
            v.push(&mut data);
            v
        });
        &mut chars[0]
    };
    *outer_reference = 'z';
    assert_eq!(data, 'z');
}

Sadly, this doesn't compile:

error[E0597]: `reusable_slice` does not live long enough
  --> lifetime-problems.rs:7:21
   |
7  |         let chars = reusable_slice.fill_mut(|mut v| {
   |                     ^^^^^^^^^^^^^^ borrowed value does not live long enough
...
11 |         &mut chars[0]
   |         ------------- borrow later used here
12 |     };
   |     - `reusable_slice` dropped here while still borrowed

There is one notable difference in the code (except for using mutable references): I had to explicitly provide the type &mut char for outer_reference because the compiler complained, but I'm not sure whether this has anything to do with the problem.

My questions are:

Why does changing &T to &mut T change the lifetimes in this case?

And more importantly: how can I get my example to compile (and work correctly)?

The source code of the crate is here: https://github.com/mgeier/rsor

Here is a branch that includes the broken example (and the working one as well) as a doctest: https://github.com/mgeier/rsor/tree/lifetime-example-broken
The error can be reproduced by running cargo test.

1 Like

I've only had a quick look, but it's almost certainly variance.

(My first instinct was variance too.)

let chars: &mut [&mut char] = ...
chars[0]

Shared references are Copy. Call to index borrows from the slice of chars. With mutable that must be held to stop any potential further modification happening.

1 Like

Thanks a lot!
That explains the difference between the mutable and the immutable case, but is there something I can do to work around this?

In the meantime, I've come up with a very similar example that doesn't involve my crate, so it might illustrate the problem better (and it shows me that this particular error is not caused by my crate!):

fn main() {
    let mut data = 'a';
    let outer_reference: &mut char = {
        let mut v = Vec::new();
        v.push(&mut data);
        let chars: &mut [&mut char] = v.as_mut();
        chars[0]
    };
    *outer_reference = 'z';
    assert_eq!(data, 'z');
}

This produces the same compiler error as I mentioned above:

error[E0597]: `v` does not live long enough
 --> lifetime-problems.rs:6:39
  |
6 |         let chars: &mut [&mut char] = v.as_mut();
  |                                       ^ borrowed value does not live long enough
7 |         chars[0]
  |         -------- borrow later used here
8 |     };
  |     - `v` dropped here while still borrowed

Is this simply impossible?

My thinking is: I don't use the outer reference anymore, so I don't care if v is dropped. I'm only interested in the inner reference (&mut char), which references the local variable data that's still in scope (and should have nothing to do with v anymore?). I don't see a problem with that, why does the compiler see one?

To phrase my question differently: If I have a &'a mut [&'b mut T] and I'm only interested in the inner references, is there a way to get rid of the outer slice (including the lifetime 'a) and use the inner references (with lifetime 'b) when the outer reference is already out of scope?

Hmm, it seems that the issue is not the lack of covariance, but the lack of a Copy impl for mutable references.

No. Generally whenever you have a construct of the form &'a mut &'b mut T, you can only use that to obtain an &'a mut T, whereas with immutable references you can copy the inner reference and obtain an &'b T.

The reason for this restriction is that removing it makes it possible to evade the uniqueness guarantee for mutable references.

2 Likes

Without unsafe in your basic example this works.

let chars: Box<[&mut char]> = v.into();

Need the owned slice to perform boxes black magic (DerefMove or maybe it is IndexMove.)

That works because you take the mutable reference out of the collection (Box) rather than reborrow from it. If you had called remove on the vector, it would work too.

Nested borrows and reborrowing

When dealing with nested borrows, of the form:

  • &'short &'long T for the shared case,

  • &'short mut &'long mut T for the unique case,

  • (and the &&mut and &mut& combinations),

we know we have a borrow over T valid for the lifetime 'short, and with the shared-ness / unique-ness resulting from combining both borrows (&& β†’ &, &mut &mut β†’ &mut, &&mut β†’ &, &mut & β†’ &; "flattened borrow is unique if all the elems of the borrow chain are unique, otherwise we must loosen the result to a shared borrow").

This leads us to "flattening" these borrows as &'short mut (for the &'short mut &'long mut case) and &'short (otherwise). This process is called reborrowing, and is the only mechanism common to both shared and unique references that allows the code to remain correct / sound.

Sadly, this can be over-constraining in the case where we'd like to extract something with the 'long-er lifetime.

Getting a 'long borrow out of a nested 'short ('long …) one

  • && β‡’ Copy

    &'short &'long T β†’ &'long T Just Worksβ„’ thanks to type U = &'long T being Copy (so we can go from &'short U β†’ U);

    But that does not work for the &mut / exclusive case, since it would otherwise be possible to get multiple exclusive references, which is absurd, and thus unsound:

  • &mut &mut β‡’ mem::replace()

    Generally, the &mut equivalent to go from &mut U β†’ U is mem::replace, which is based on being able to take some value of type U by putting another one in its stead.

    Variants of these are:


Application

So, if we go back to your reduced example:

fn main() {
    let mut data = 'a';
    let outer_reference: &mut char = {
        let mut v = Vec::new();
        v.push(&mut data);
        let mut chars: &mut Vec<&mut char> = &mut v;
        chars.swap_remove(0)
    };
    *outer_reference = 'z';
    assert_eq!(data, 'z');
}

And for your general use-case, you'd need to do something like either yielding a &mut Vec<…> instead of a &mut […] so that they can .drain() / .swap_remove() themselves, or a &mut [Option<&mut _>], so that they can .take().unwrap() the [i]-th value.

11 Likes

Wow, thank you for this elaborate explanation @Yandros! :heart:
I really like your outline of the different mem-methods and the analog variants for Vec
I have put your answer into my bookmarks. :bookmark:

2 Likes

Wow indeed. Thanks for this enlightening answer!

In my use case, I don't actually want to expose my data structure, I just want to give access to its contents by returning a (mutable) slice of references.

But I think std::mem::replace() is the answer!

I've come up with this modified example, which compiles and works:

let mut data = 'a';
let mut dummy = 'x';
let outer_reference = {
    let mut v = Vec::new();
    v.push(&mut data);
    let chars: &mut [&mut char] = v.as_mut();
    std::mem::replace(&mut chars[0], &mut dummy)
};
*outer_reference = 'z';
assert_eq!(data, 'z');

The same thing works with my rsor::Slice data structure from my original example.

This is of course a totally contrived example, and I'm not planning to actually ever use anything like this. But it confirms that different lifetimes are actually possible in a mutable slice of references, which is all I wanted to know.

Thanks to all participants for their answers!

2 Likes
But unlike normal traits, we can use them as concrete and sized types, just like structs.

Now, say we have a very simple function that takes an Animal, like this:


fn love(pet: Animal) {
    pet.snuggle();
}

That is talking about tait objects, right? Shouldn't it be Box<dyn Animal> or impl Animal? I didn't think you could pass a trait directly like that.

It doesn't say what makes Animal "unlike normal traits".

You can't. The explanation is a few paragraphs earlier:

To keep things simple, this section will consider a small extension to the Rust language that adds a new and simpler subtyping relationship. After establishing concepts and issues under this simpler system, we will then relate it back to how subtyping actually occurs in Rust.

The language being described on that page isn't Rust, but a rhetorical language that has support for non-lifetime-based subtyping (similar to subclassing in languages that support that). The code doesn't actually work, and never will; it's just meant to explain what is meant by "variance".

It's probably written that way because a lot of people are familiar with subtyping relationships from languages like Java, Python, etc. and their application to lifetimes isn't immediately obvious, so it's easier to start with a concept the reader already knows. Still, I agree it's a bit awkward.

1 Like