Multiple mutable methods

Heres one way to see how lifetime analysis is working.

First the definition of Foo

struct Foo<'a> {
    lookup: HashMap<u32, Bar<'a>>,
    current: Option<&a' Bar<'a>>
}

Here we are saying that the lifetime in current is the same as in lookup. Simple stuff, binding lifetimes together.

Next, in Foo::new

    ...
    fn new() -> Self {
        let mut lookup = HashMap::new();
        lookup.insert(42, Bar{message: "secret to life, the universe, and everything"});
        
        Self{
            lookup,
            current: None
        }
    }
    ...

Because string literals have the type &'static str, they subtype &'a str. So it is safe to use &'static str where &'a str is expected (in lookup). So now you can create an arbitrary Foo<'a>.

Finally on to calc

    fn calc(&mut self) -> &str {
        if self.current.is_none() {
            self.current = Some(self.lookup.get(&42).unwrap());
        }
        self.current.unwrap().message
    }

Here the important thing to note is the assignment to current. This is important because the type of current is &'a Bar<'a> (remember this, as it will be important shortly).

Also note that we have self behind a &mut, this is important because &mut T is invariant over T. What this means for lifetimes is that the &mut &'a T is only equal to &mut &'b T if and only if 'a == 'b (ignoring the lifetimes associated with the &'0 mut, labeled '0 here).

What this means for current is that, this assignment will determine the lifetime 'a.

Now, that we got that, let's desugar calc.

    fn calc<'b>(&'b mut self) -> &'b str {
        if self.current.is_none() {
            self.current = Some(self.lookup.get(&42).unwrap());
        }
        self.current.unwrap().message
    }

We now have some other lifetime 'b. Now let's examine the assignment. we are assigning to current, what are we assigning. If we look at the RHS we see, Some(self.lookup.get(&42).unwrap()), which is of the type Option<&'b Bar<'a>>. But we are trying to assign it to the LHS, current, which has the type Option<&'a Bar<'a>>.

Oh no! What if 'a: 'b, and 'a != 'b. This would mean that this assignment is invaid! It would produce a dangling pointer. Hence why &mut T is invariant over T. Because of this, Rust checks that 'a == 'b, but it can't enforce this in general because you could pick 'b to be any lifetime that you want.

So if we correct this signature, and enforce that &'a mut self, then 'a == 'b, and everything works out.

So

    fn calc(&'a mut self) -> &'a str {
        if self.current.is_none() {
            self.current = Some(self.lookup.get(&42).unwrap());
        }
        self.current.unwrap().message
    }

works. Now you may notice that I added a lifetime to &'a str. Rust does this desugaring automatically, it is called lifetime elision, and it makes working with lifetimes painless in the normal case, and it is what your lifetime annotated code desugars to :slight_smile: .

Note that this also works

    fn calc<'b>(&'b mut self) -> &'b str
    where 'b: 'a {
        if self.current.is_none() {
            self.current = Some(self.lookup.get(&42).unwrap());
        }
        self.current.unwrap().message
    }

I will leave this to you to figure out why.


edited because I was wrong in my previous atempt at this post, lifetimes are hard and I missed the actual problem with the code. I aslo didn't consider variance so that led me astray. But this is the correct reason for why calc didn't compile without annotation.

edit 2: formatting and wording

8 Likes

Wow... okay... starting to get it a little more! Not quite there, but closer :slight_smile:

OK... two questions:

  1. How is RHS typed as Option<&'b Bar<'a>> ? i.e. if self is &'b mut self, where is the 'a coming from... wouldn't it be Option<&'b Bar<'b>> ?

  2. Similarly, since self is defined as &'b mut self, isn't the LHS current also Option<&'b Bar<'a>> (or Option<&'b Bar<'b>>)?

I know it's almost exactly speaking to what you said here:

But I don't quite get how that's true if self is &'b mut self

Man it would be so nice if someone made like an interactive lifetime analysis tutorial thing

For example, in something like the style of the demo under the "Who's Right" section on this JS tasks post

1 Like

The reason for that is because of the signature for HashMap::<K, V>::get which is fn<'c, Q>(&'b self, &Q) -> Option<&'c V> where Q: Borrow<K>. Effectively the gives you a reference with the same lifetime as self but it doesn't change the lifetime inside V. So applying this to our example, we find that the concrete HashMap we are using is HashMap<u32, Bar<'a>> because that is the type of lookup. This means that the concrete HashMap::get function has the signature fn<'c, Q>(&'c self, &Q) -> Option<&'c Bar<'a>> where Q: Borrow<u32>. Note we have to copy and paste V because that is how generics work, so we can't change the lifetimes inside of V. This is why we get Option<&'b Bar<'a>> for the type of the RHS.

The type of current depends of Self not &'b mut Self. So no, current's type does not depend on 'b. If this was allowed, then current would change types depending on where it was called. Which doesn't match our intuition on types.


I think the confusion is coming from liferimes infecting everything. But this isn't how lifetimes work. Think of lifetimes as generic parameters along the lines of generic type parameters and this will start to make more sense.

1 Like

Okay - think I'm going to print up this page and sit with it over a cup of tea till I understand it, or at least before responding with my next bit of confusion :wink:

Definitely! In fact I even caught myself holding onto the idea of the lifetime affecting Foo as a whole instead of the individual objects it holds

1 Like

@RustyYato's explanation was great, I'll just add a comment on the following pattern, when looking at it with some "distance":

struct Foo<'values> { // I like explicitly named lifetimes
    lookup: HashMap<u32, Bar<'values>>,
    current: Option<&'values Bar<'values>>,
}

impl<'values> Foo<'values> {
    fn calc (&'values mut self) -> &'values str
    {
        if self.current.is_none() {
            self.current = Some(self.lookup.get(&42).unwrap()); // Self referential
        }
        self.current.unwrap().message
    }

This patterns creates a self-referential struct (hence the 'values lifetime requirement on the &mut self borrow, explained by @RustyYato), which in practice prevents the programmer from using that struct ever again (the main reason you may be having these borrowing issues):

  • Foo<'values> cannot live longer than the lifetime of the values it contains, 'values.

  • fn calc (&'values mut self, ...) -> ... this borrows mut-ably (i.e., in a unique / non-sharing fashion) your Foo<'values> for the whole 'values lifetime.

Since the struct cannot live longer than 'values, and since it is mut borrowed during the whole 'values lifetime, it is mut borrowed until it dies. And since a mut borrow is exclusive (the very definition of a mut borrow), after calling the .calc(...) method, it is impossible to access the struct ever again.

The solution? Do not use lifetimes / borrows for interior references.

In your case, you could just have:

struct Foo<'values> {
    lookup: HashMap<u32, Bar<'values>>,
    current: Option<u32>, // index / key instead of a reference
}

impl<'values> Foo<'values> {
    fn calc (&'_ mut self) -> Option<&'_ str> // anonymous lifetime can be shorter than `'values`
    {
        if self.current.is_none() {
            self.current = Some(42); // the line that constrained `'_` to be as big as  `'_ values` is no longer here
        }
        self.lookup
            .get(self.current.unwrap())
            .map(|bar| bar.message)
    }

     /// or more simply (no longer requiring mutation)
    fn calc (&'_ self) -> Option<&'_ str> // anonymous lifetime can be shorter than `'values`
    {
        self.lookup
            .get(self.current.unwrap_or(42))
            .map(|bar| bar.message)
    }
3 Likes

In fact, the reason I used std::cell::Cell in my example of self-referential types is because of what @Yandros said. By using Cell I can avoid the mutable reference, and thus uniquely borrowing the type and so I can still use it later, even if I can't move it after using the self-referential method.

1 Like

So - as sortof a tangent, I was thinking about this... and it seems like a really good idea even if just as a teaching tool... for example, perhaps reworking some of the above examples but with the following named lifetimes would be beneficial:

'instance: what we used for 'a, meaning the lifetime of a newly created Foo
'all: what we used for 'b, meaning every possible lifetime (e.g. forall 'b)
'instance2: what we used for 'c', meaning the lifetime of a newly created HashMap return value

Maybe not those exact names / descriptions, but I think this is a good approach to teaching lifetimes, at least sometimes.

I wonder why that is, actually, since with regular generics it's usually easier to keep them completely generic... i.e. it's more useful to think of map as like (A -> B) -> F A -> F B than something like (TYPE1 -> TYPE2) ... because generic functions really don't want to know anything about those types (putting aside traits for the moment) ... but lifetimes have inherent meaning... like they're tied to something else outside the function..?

Not sure if I'm explaining my thoughts there well, but whatever - might inspire some useful feedback :slight_smile:

Hmmmm.... I'm trying to optimize early (I know..) and avoid a HashMap lookup where it's not needed, but most likely - it's probably an unnecessary optimization overall. (e.g. I may not really need to do multiple lookups as much as I think). Reminds me a little of what @vorner wrote here:

"The difference between good and bad design is visible in the hairiness of API"

More food for thought!

I'm not quite following where 'c is coming from here... is it automatically gotten from Q ?

And if so - the reason we're saying 'c == 'b is because the locally scoped value we gave it, &42, is 'b?

Not following why this is... doesn't the mut borrow finish when calc() ends?

EDIT: okay... I think I see... by annotating a lifetime I'm explicitly telling it that it does not finish borrowing when calc() ends... but rather when the lifetime ends... that's funky!

2 Likes

So, continuing with the learning process - it seems thinking in terms of lifetimes is the better mental model for borrowing overall, i.e. not so much literal scope blocks?

So taking this example from The Rust Book:

let mut s = String::from("hello");

{
    let r1 = &mut s;

} // r1 goes out of scope here, so we can make a new reference with no problems.

let r2 = &mut s;

We could think of it in terms of lifetimes more like this (just for reference, won't compile):

fn main() { //LIFETIME: 'a
    
    let mut s = String::from("hello");

    { //LIFETIME: 'b
        let r1 = &'b mut s;
    } // 'b is killed here

    let r2 = &'a mut s;
} //'a is killed here

There's no practical difference here, but then this opens the door for understanding that we can tell the compiler what the lifetime of a borrow is - so if we could tell the compiler here that r1 is 'a, explicitly, we'd get the same error as not having the block - even if we were in the block...

Added another post but it's really a separate subject, will open a new thread..

FYI I have no idea where I picked it up :wink:

FYI, I think the following might be a simpler example to see the original problem (after my silly first mistake) since it avoids lifetimes and hashmaps in the struct definition.

If I'm not mistaken, it's the same exact issue when it comes down to it - would really appreciate it if someone would confirm:

Playground

struct Foo {
    bar: String,
    baz: String
}


impl Foo {
    fn new() -> Self {
        Self {
            bar: String::from("hello"),
            baz: String::from("world")
        }
    }
    
    fn get_bar_mut(&mut self) -> &mut str {
        self.bar.as_mut_str()
    }
    
    fn get_baz_mut(&mut self) -> &mut str {
        self.baz.as_mut_str()
    }
}


fn main() {
    let mut foo = Foo::new();
    
    //this is fine
    let bar = foo.get_bar_mut();
    println!("{}", bar);
    let baz = foo.get_baz_mut();
    println!("{}", baz); 
    
    //this is not
    let bar = foo.get_bar_mut();
    let baz = foo.get_baz_mut();
    println!("{} {}", bar, baz);
}

Here's an annotated and commented version:

Playground

struct Foo {
    bar: String,
    baz: String
}


impl Foo {
    fn new() -> Self {
        Self {
            bar: String::from("hello"),
            baz: String::from("world")
        }
    }
    
    fn get_bar_mut<'b> (&'b mut self) -> &'b mut str {
        self.bar.as_mut_str()
    }
    
    fn get_baz_mut<'c> (&'c mut self) -> &'c mut str {
        self.baz.as_mut_str()
    }
}


fn main() {
    let mut foo = Foo::new();
    
    //'b begins here
    let bar = foo.get_bar_mut();
    println!("{}", bar);
    //'b ends here
    
    //'c begins here
    let baz = foo.get_baz_mut();
    println!("{}", baz); 
    //'c ends here
    
    //No problems so far - each lifetime has one mutable borrow
    //However....
    
    //'b begins here
    let bar = foo.get_bar_mut();
    
    //'c begins here
    let baz = foo.get_baz_mut();
    
    //we use both here - so neither lifetime has ended yet
    println!("{} {}", bar, baz);
    
    //'b and 'c end here
    //uhoh - &self is borrowed mutably twice within the span of 'b
    //And there is no way to work around this with blocks since
    //both are used at the same call site
}

IF that's all correct... then what's the idiomatic way to deal with this? I'm sure it comes up often...

Decoupling "bar" and "baz" is not feasible - it's really all too tightly related and should travel together

However the mutability does not need to happen from the outside. Only within the get_bar_mut() and get_baz_mut() functions (for example mutating something other than bar/baz on self, before returning bar/baz)

Spelling that out - I'm thinking that maybe the idiomatic answer is to make Foo's API be completely immutable, and store bar/baz via RefCell for interior mutability?

(side-question - not completely sure whether I need RefCell or if Cell will suffice - but I know there's good explanations of that somewhere)

If "bar" and "baz" is tightly coupled, how about this?

fn get_bar_and_baz_mut(&mut self) -> (&mut str, &mut str) {
  (&mut self.bar, &mut self.baz)
}
1 Like

Good tip for this specific scenario, thanks!

In my actual project it might not work out exactly like that, but I'll keep it in mind as another idea

So far refactoring to use interior mutability seems to be much nicer all around

ugh nevermind... might go back to &mut self and use the tuple idea afterall...

Forgot about this... but coming back to it, is it that the function returns 'b, and it's defined as a method of Self with lifetime 'a - so all returned references must come from either an outside source or outlive 'a ?

(i.e. the only places we can get a lifetime from are from input parameters or Self - since this function doesn't take any input parameters besides Self, the only thing we have to derive from is 'a, and therefore 'b must outlive that)

It's still a little hard for me to see how we can have 'b as a lifetime for self... like, isn't Self always 'a?

I think I asked this before but it's still bothering a bit... I mean I can accept it in the sense of "it's just a generic on a function".... but it feels so weird to define a lifetime on Self, and then say "no, actually it's a different lifetime" when accepting &self in the method.

Note that the lifetime 'b is not on Self, but &'b mut Self. Important distinction.


One thing to note, there is an implicit bound at play, and then rest falls out of that

Copied from my previous answer

fn calc<'b>(&'b mut self) -> &'b str
where 'b: 'a {
    if self.current.is_none() {
        self.current = Some(self.lookup.get(&42).unwrap());
    }
    self.current.unwrap().message
}

The hidden bound is Self: 'b. Which is necessary to prevent dangling pointers.
If we add that in we get the signature

fn calc<'b>(&'b mut self) -> &'b str
where 'b: 'a, Self: 'b

We can expand this to

fn calc<'b>(&'b mut self) -> &'b str
where 'b: 'a, Foo<'a>: 'b

Now Rust knows that if Foo<'a>: 'b, then 'a: 'b, so we get this

fn calc<'b>(&'b mut self) -> &'b str
where 'b: 'a, Foo<'a>: 'b, 'a: 'b

But look, we now have the two rules that 'a: 'b and 'b: 'a, which means that 'a == 'b so calc reduces to

fn calc(&'a mut self) -> &'a str

Which we already analyzed

4 Likes

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.