I thought I got a somewhat decent mental model about the borrow rules and lifetimes, but I stumbled upon this code snippet and now I am not sure anymore I actually understand.
The example:
use std::marker::PhantomData;
struct Handle<'a> {
_fake: PhantomData<&'a ()>,
}
fn make_handle<'a>(_: &'a mut [u32]) -> Handle<'a> {
Handle { _fake: PhantomData }
}
fn main() {
let mut od = [0u32; 10];
let handle_1 = make_handle(&mut od);
// let handle_2 = make_handle(&mut od); // FAIL: cannot borrow `od` as mutable again
drop(handle_1);
}
This fails because the od is mutably borrowed a second time, and that's a no-go according to the mutable aliasing rule (not allowing multiple mutable borrows). Of course, I can see - visibly - that there are 2 mutable borrows: that - per definition - would break the rule.
But what trips me up is: the Handle that is returned from the make_handle() function does not have any actual "relation" to the od, so why is this &mut od mutable borrow still "alive" (and thus conflict with the second mutable borrow)? I was inclined to think that after the first function call to make_handle, this reference would not be borrowed anymore, so we could just go ahead with borrowing od mutably a second time (i.e., isn't this also what non-lexical lifetimes are about?).
After trying out some things and searching for a good explanation (online), it seems (at least, according to my own summary of my investigation) that the lifetime 'a of the phantom reference _fake that is tied to the lifetime of the od (but is not even a reference to the od!), makes the compiler decide that the &mut od should keep on existing, even after the function call to make_handle. For me, this feels somewhat strange that merely the lifetime usage seems to "extend" the borrow of the od, while there is no "real" connection between handle_1 and od (i.e., there is no real reference in handle_1 to the od). How should I see this; what am I missing or misunderstanding here?
(I did make one observation that does make sense IMHO: if the borrow of the first &mut odwould end after the function call (like I assumed), the returned Handle would not be able to live any longer than exactly after returning from the function call as its lifetime of that Handle is tied to that mutable borrow of the od. But I am not sure this is the correct and complete picture.)
In contrast, the following example does make sense in my mental model:
fn get_mut_value<'a>(array: &'a mut [u32], index: usize) -> &'a mut u32 {
&mut array[index]
}
fn main() {
let mut od: [u32;10] = [0;10];
let handle_1 = get_mut_value(&mut od, 5);
// let handle_2 = get_mut_value(&mut od, 6); // fails, no second borrow allowed
println!("{}", handle_1);
}
Here, handle_1 has an actual relationship with the od array: it is referencing inside the actual od object, so please don't hand out a second mutable borrow because handle_1 is still used later: OK, makes sense. Put differently, I always thought that the second borrow was disallowed because the returned item has an actual reference in the array, but maybe here the reason is also rather the shared lifetime of the returned item (and not really the fact that there is reference in the returned item) as in the first example?
Could anyone give it a go to explain to me what is going on?
struct Handle<'a> {
// some fields
}
fn make_handle<'a>(_: &'a mut [u32]) -> Handle<'a> {
// some impl
}
to the compiler, they are the same, and will behave the same (except for auto-trait and variance, the compiler will look at the fields)
the compiler definitely wil not look inside the implementation except for checking that it is correct. how you implement fn make_handle<'a>(_: &'a mut [u32]) -> Handle<'a> is up to you, as long as it respect the contract its signature has made.
the contract says the Handle<'a> must live as long as the &'a mut, so that's what will happen.
What is a “real” connection, anyway.. Note that PhantomData-fields do come up similarly in real handle types, too. Like… if you look at slice::IterMut<'a>. When a value of that type is constructed the raw-pointer fields are not in any way “understood” by the borrow checker, actually. The only thing that makes the borow checker understand what’s going on is really just that the relevant, and unsafe{}-ly implemented API was simply defined with a type signature of fn …<'a>(&'a mut [T]) -> IterMut<'a> that used the same generic lifetime parameter in two places. That’s all that establishes the connection to for borrow checking in all usage sites of this kind of API (indirectly then, through the .iter_mut() method of [T]).
Yes, this is accurate. From the compiler’s POV, only the function signature matters, not what’s inside.
One good way to notice this would also be to replace the implementation of get_mut_value with something like panic!(). The borrow-checking at the call site won’t change from this, at all.
From a practical POV of course, the only reason to (intentionally) write a function signature like this in the first place is because the returned item has an actual reference; or something morally equivalent to an actual reference.
The issue remains of course, that if one unintentionally declares an overly restrictive (to the caller) function signature on some API, then the result can be confusing compilation errors, because they’ll make claims about the usage site that may seem weird / inaccurate, and the actual bug was at a different point (the wrongly function signature with over-restrictive lifetime bounds / relationships).
As another resource, I also like this video, containing some nice examples to look at when further developing your mental model of lifetimes & borrowing rules:
on the subject of variance. (that's a pretty complex subject so if you'r not familiar, you might want to skip this bit until you have learned quite a bit about it.)
you can make your example code work by making Handle contravariant in 'a :
use std::marker::PhantomData;
struct Handle<'a> {
_fake: PhantomData<fn(&'a ())>,
}
fn make_handle<'a>(_: &'a mut [u32]) -> Handle<'a> {
Handle { _fake: PhantomData }
}
fn main() {
let mut od = [0u32; 10];
let handle_1 = make_handle(&mut od);
let handle_2 = make_handle(&mut od); //Works because `Handle` is contravariant over its lifetime
drop(handle_1);
}
"The input to this function is still borrowed while return is in use."
I think that's one of the things I'm struggling with. What I think you mean:
{
let x = 5;
let y = foo(&x);
// do something with y
// here &x is still borrowed?
}
Why is &x still borrowed until the end of the scope? Is that because that's per definition of the lifetime of a reference borrow: that lasts until the end of the scope, even though that &x is not used anywhere else anymore in that same scope, or is this not true?
Your misconception is assuming that the compiler tracks what really happens.
Rust doesn't have a garbage collector. It doesn't track the real-world state of borrowing/aliasing.
Borrow checking is a purely abstract game, played using its own rules.
The lifetime annotations don't have to be backed by anything justifying them. You can make something be "borrowed" as a pure fiction, the same way you can make struct non-copyable and have the compiler enforce it, even though everything in RAM is copyable.
The compiler will check whether your annotations are consistent and don't violate the rules, but it's all done abstractly. It's like chess where you "can't" move a piece in the game, even if physically you can.
This can be used to make types that enforce relationships and exclusivity through the type system even for things that aren't related to memory management directly, e.g. you can have zero-sized types "borrowed" as tokens to enforce that certain functions must be called in a specific order.
The borrows don't need to last until the end of scope (that is what NLL brought us). A lifetime will be alive so long as values with that lifetime in its type are in use, or other lifetimes cause it to be alive due to outives constraints ('a: 'b).
There are some cases where borrows can end before the associated lifetime ends,[1] such as when a reference is overwritten. And defining when a value is "in use" can also be nuanced due to destructors which may or may not be considered to be able to observe their associated borrows.
But as far as simple examples go, such as the ones in this thread so far -- you can tell the borrow of x ends when there are no more uses of y, the borrow of od stayed alive in the OP because there were later uses of handle_1, etc.
Thanks, this remark (and other parts of your answer) confirmed more or less what I was thinking (no other answer seemed to confirm this particular part, or I missed it), but I didn't really understand why/how it was keeping it alive. Especially, because I was under the assumption the compiler also looked inside the function and would need a "real" connection (e.g. an actual reference) to the od (and there was none in this example), I was not understanding why it would keep the od alive.
Based on some of the other answers in the thread, which pointed out that the function signature alone is what matters, and after consulting additional resources (such as the Foo and fn mutate_and_share<'a> example on this Rustonomicon page and some older problem cases mentioned in the RFC of non-lexical lifetimes) I now understand that the lifetime annotation of the return value can keep the input borrow “alive”.
I haven't fully dived into the variance concept, but this SO answer does summarize it a bit (with a useful analogy) - in short - to that the input argument should outlive the return value. (Of course, I can't really confirm that those takeaways are correct, it does seem to make sense. )
... Combine these two points and the behavior of a function fn foo(&'a T) -> &'a T makes sense: the value passed to the argument must be a subtype of &'a T , so it must live at least as long as 'a . The value returned can only be used as a supertype of &'a T , so it can only be assumed to live as long as 'a . Together we get that the argument outlives the return value.
I simplified my examples, but actually the PhantomData one originated from source code resembling the IterMut<'a>: thanks for the additional reference.
One good way to notice this would also be to replace the implementation of get_mut_value with something like panic!() . The borrow-checking at the call site won’t change from this, at all.
One more question: while I tend to agree to what you're saying (as I understood it the same way), it seems to contradict more or less what is written here in the book:
Lifetime annotations don’t change how long any of the references live. [...]
Because now we seem to conclude that because of this lifetime annotation for the return value, it actually changes how long the array reference lives, further along in the program.
This is a subtle difference indeed, but no, the book is right. It's usually said as "lifetimes are descriptive, not prescriptive" - that is, they describe the relations that already exist in the code, but not change them. What you are referring to is the fact that these descriptions can deviate from the actual behavior (which usually leads to contradiction and compile error), but if the code compiles, then lifetimes doesn't affect the result.
I think the book is rather confusingly written here. The start of the page introduces the following notion:
One detail we didn’t discuss in the “References and Borrowing” section in Chapter 4 is that every reference in Rust has a lifetime, which is the scope for which that reference is valid.
The notion of “valid” here is generally referring to previously mentioned principles, e.g. in a previous chapter already it came up in
Let’s recap what we’ve discussed about references:
At any given time, you can have either one mutable reference or any number of immutable references.
References must always be valid
In the next sentence(s) then,
Most of the time, lifetimes are implicit and inferred, just like most of the time, types are inferred. We are only required to annotate types when multiple types are possible. In a similar way, we must annotate lifetimes when the lifetimes of references could be related in a few different ways.
it sound like “lifetimes” refers directly to the (often-inferred) kind of information that the borrow checker can determine and understand.
In the section after, this is seemingly further confirmed
This section does particularly strongly mix up different notions of “lifetime” under the same name, since the image depicts lexical scope. It also uses the term of “live(s)” (lives longer; does not live as long; … etc.)
The term “live(s)” first came up in the section before that, which I guess we’d also have to look at then. That section contains some first explanations of the compiler error for this same code example; which used “live” in it’s phrasing of “x does not live long enough”
The error message says that the variable x “does not live long enough.” The reason is that x will be out of scope when the inner scope ends on line 7. But r is still valid for the outer scope; because its scope is larger, we say that it “lives longer.” If Rust allowed this code to work, r would be referencing memory that was deallocated when x went out of scope, and anything we tried to do with r wouldn’t work correctly. So, how does Rust determine that this code is invalid? It uses a borrow checker.
This section in the book also talks about how the reference r“is still valid” for the outer scope (which seems somewhat different than the notion of “valid” used elsewhere) and instead calls the entire code “invalid”. E.g. the previously quoted subsequent section puts it as follows in its introduction and conclusion:
The Rust compiler has a borrow checker that compares scopes to determine whether all borrows are valid.
[…]
Now that you know where the lifetimes of references are and how Rust analyzes lifetimes to ensure that references will always be valid, let’s explore generic lifetimes in function parameters and return values.
I wasn’t even aiming to complain about these different section though, just provide the context to understand where and how “lifetime” and “live(s)” comes up, so we can try to make sense of this statement:
Lifetime annotations don’t change how long any of the references live. Rather, they describe the relationships of the lifetimes of multiple references to each other without affecting the lifetimes. […]
I find “live(s)” still a rather unclear term being used here. I think it’s least confusing in statements such as “The program is rejected because […] [t]he subject of the reference doesn’t live as long as the reference.” but I’m not even sure whether or not “how long X lives” is meant to be distinct from “lifetime of X” in this page of the book, or perhaps that’s even inconsistently only sometimes the case
I can imagine multiple valid facts that this statement could be referring to, anyway. One would be that the lifetime annotations “don’t change how long any of the references live” because borrow checking never really changes anything, in can only reject certain programs; which are rejected when the compiler “analyzes lifetimes” and then concludes it is not sure that all references are always valid.
Another reading could be that “how long the reference lives” is a different concept from “lifetime of the reference”, the latter being instead a measure (or rather lower bound) on how long the thing being referenced will live for.
(On that note, it’s probably also unfortunate that the section before “lifetime annotation syntax” already uses a kind of metasyntax for “lifetime” that looks completely identical, using names like 'a and 'b, even though it arguably refers to subtly different things than the actual function of the (always generic) lifetime annotations in function signature (and some other places) in actual Rust syntax & programs.)
In any case, I think this sentence doesn’t really help explain anything on its own.
And there’s a more in-depth coverage of much more relevant insights anyway; the statement here simply seems redundant and not helpful, without further explanation. For example at a later point it states this:
When annotating lifetimes in functions, the annotations go in the function signature, not in the function body. The lifetime annotations become part of the contract of the function, much like the types in the signature. Having function signatures contain the lifetime contract means the analysis the Rust compiler does can be simpler. If there’s a problem with the way a function is annotated or the way it is called, the compiler errors can point to the part of our code and the constraints more precisely. If, instead, the Rust compiler made more inferences about what we intended the relationships of the lifetimes to be, the compiler might only be able to point to a use of our code many steps away from the cause of the problem.
Which is really important, and the way in which the annotation does seem to “affect” things, because it defines a contract, and the compiler then reasons locally based on that contract. Also the fact that the borrow checker really is a checker, and only aims to “ensure that references will always be valid” correctly, not aims to be complete in the sense of understand every single program with this property (references always valid), could probably be highlighted better, because it’s the core reason why it’s completely expected behavior that a bad lifetime signature – an incorrectly defined contract – will make the compiler reject an otherwise “correct” program.
)
