I thought I got a somewhat decent mental model about the borrow rules and lifetimes, but I stumbled upon this code snippet and now I am not sure anymore I actually understand.
The example:
use std::marker::PhantomData;
struct Handle<'a> {
_fake: PhantomData<&'a ()>,
}
fn make_handle<'a>(_: &'a mut [u32]) -> Handle<'a> {
Handle { _fake: PhantomData }
}
fn main() {
let mut od = [0u32; 10];
let handle_1 = make_handle(&mut od);
// let handle_2 = make_handle(&mut od); // FAIL: cannot borrow `od` as mutable again
drop(handle_1);
}
This fails because the od is mutably borrowed a second time, and that's a no-go according to the mutable aliasing rule (not allowing multiple mutable borrows). Of course, I can see - visibly - that there are 2 mutable borrows: that - per definition - would break the rule.
But what trips me up is: the Handle that is returned from the make_handle() function does not have any actual "relation" to the od, so why is this &mut od mutable borrow still "alive" (and thus conflict with the second mutable borrow)? I was inclined to think that after the first function call to make_handle, this reference would not be borrowed anymore, so we could just go ahead with borrowing od mutably a second time (i.e., isn't this also what non-lexical lifetimes are about?).
After trying out some things and searching for a good explanation (online), it seems (at least, according to my own summary of my investigation) that the lifetime 'a of the phantom reference _fake that is tied to the lifetime of the od (but is not even a reference to the od!), makes the compiler decide that the &mut od should keep on existing, even after the function call to make_handle. For me, this feels somewhat strange that merely the lifetime usage seems to "extend" the borrow of the od, while there is no "real" connection between handle_1 and od (i.e., there is no real reference in handle_1 to the od). How should I see this; what am I missing or misunderstanding here?
(I did make one observation that does make sense IMHO: if the borrow of the first &mut odwould end after the function call (like I assumed), the returned Handle would not be able to live any longer than exactly after returning from the function call as its lifetime of that Handle is tied to that mutable borrow of the od. But I am not sure this is the correct and complete picture.)
In contrast, the following example does make sense in my mental model:
fn get_mut_value<'a>(array: &'a mut [u32], index: usize) -> &'a mut u32 {
&mut array[index]
}
fn main() {
let mut od: [u32;10] = [0;10];
let handle_1 = get_mut_value(&mut od, 5);
// let handle_2 = get_mut_value(&mut od, 6); // fails, no second borrow allowed
println!("{}", handle_1);
}
Here, handle_1 has an actual relationship with the od array: it is referencing inside the actual od object, so please don't hand out a second mutable borrow because handle_1 is still used later: OK, makes sense. Put differently, I always thought that the second borrow was disallowed because the returned item has an actual reference in the array, but maybe here the reason is also rather the shared lifetime of the returned item (and not really the fact that there is reference in the returned item) as in the first example?
Could anyone give it a go to explain to me what is going on?
struct Handle<'a> {
// some fields
}
fn make_handle<'a>(_: &'a mut [u32]) -> Handle<'a> {
// some impl
}
to the compiler, they are the same, and will behave the same (except for auto-trait and variance, the compiler will look at the fields)
the compiler definitely wil not look inside the implementation except for checking that it is correct. how you implement fn make_handle<'a>(_: &'a mut [u32]) -> Handle<'a> is up to you, as long as it respect the contract its signature has made.
the contract says the Handle<'a> must live as long as the &'a mut, so that's what will happen.
What is a “real” connection, anyway.. Note that PhantomData-fields do come up similarly in real handle types, too. Like… if you look at slice::IterMut<'a>. When a value of that type is constructed the raw-pointer fields are not in any way “understood” by the borrow checker, actually. The only thing that makes the borow checker understand what’s going on is really just that the relevant, and unsafe{}-ly implemented API was simply defined with a type signature of fn …<'a>(&'a mut [T]) -> IterMut<'a> that used the same generic lifetime parameter in two places. That’s all that establishes the connection to for borrow checking in all usage sites of this kind of API (indirectly then, through the .iter_mut() method of [T]).
Yes, this is accurate. From the compiler’s POV, only the function signature matters, not what’s inside.
One good way to notice this would also be to replace the implementation of get_mut_value with something like panic!(). The borrow-checking at the call site won’t change from this, at all.
From a practical POV of course, the only reason to (intentionally) write a function signature like this in the first place is because the returned item has an actual reference; or something morally equivalent to an actual reference.
The issue remains of course, that if one unintentionally declares an overly restrictive (to the caller) function signature on some API, then the result can be confusing compilation errors, because they’ll make claims about the usage site that may seem weird / inaccurate, and the actual bug was at a different point (the wrongly function signature with over-restrictive lifetime bounds / relationships).
As another resource, I also like this video, containing some nice examples to look at when further developing your mental model of lifetimes & borrowing rules:
on the subject of variance. (that's a pretty complex subject so if you'r not familiar, you might want to skip this bit until you have learned quite a bit about it.)
you can make your example code work by making Handle contravariant in 'a :
use std::marker::PhantomData;
struct Handle<'a> {
_fake: PhantomData<fn(&'a ())>,
}
fn make_handle<'a>(_: &'a mut [u32]) -> Handle<'a> {
Handle { _fake: PhantomData }
}
fn main() {
let mut od = [0u32; 10];
let handle_1 = make_handle(&mut od);
let handle_2 = make_handle(&mut od); //Works because `Handle` is contravariant over its lifetime
drop(handle_1);
}
"The input to this function is still borrowed while return is in use."
I think that's one of the things I'm struggling with. What I think you mean:
{
let x = 5;
let y = foo(&x);
// do something with y
// here &x is still borrowed?
}
Why is &x still borrowed until the end of the scope? Is that because that's per definition of the lifetime of a reference borrow: that lasts until the end of the scope, even though that &x is not used anywhere else anymore in that same scope, or is this not true?
Your misconception is assuming that the compiler tracks what really happens.
Rust doesn't have a garbage collector. It doesn't track the real-world state of borrowing/aliasing.
Borrow checking is a purely abstract game, played using its own rules.
The lifetime annotations don't have to be backed by anything justifying them. You can make something be "borrowed" as a pure fiction, the same way you can make struct non-copyable and have the compiler enforce it, even though everything in RAM is copyable.
The compiler will check whether your annotations are consistent and don't violate the rules, but it's all done abstractly. It's like chess where you "can't" move a piece in the game, even if physically you can.
This can be used to make types that enforce relationships and exclusivity through the type system even for things that aren't related to memory management directly, e.g. you can have zero-sized types "borrowed" as tokens to enforce that certain functions must be called in a specific order.
The borrows don't need to last until the end of scope (that is what NLL brought us). A lifetime will be alive so long as values with that lifetime in its type are in use, or other lifetimes cause it to be alive due to outives constraints ('a: 'b).
There are some cases where borrows can end before the associated lifetime ends,[1] such as when a reference is overwritten. And defining when a value is "in use" can also be nuanced due to destructors which may or may not be considered to be able to observe their associated borrows.
But as far as simple examples go, such as the ones in this thread so far -- you can tell the borrow of x ends when there are no more uses of y, the borrow of od stayed alive in the OP because there were later uses of handle_1, etc.
Thanks, this remark (and other parts of your answer) confirmed more or less what I was thinking (no other answer seemed to confirm this particular part, or I missed it), but I didn't really understand why/how it was keeping it alive. Especially, because I was under the assumption the compiler also looked inside the function and would need a "real" connection (e.g. an actual reference) to the od (and there was none in this example), I was not understanding why it would keep the od alive.
Based on some of the other answers in the thread, which pointed out that the function signature alone is what matters, and after consulting additional resources (such as the Foo and fn mutate_and_share<'a> example on this Rustonomicon page and some older problem cases mentioned in the RFC of non-lexical lifetimes) I now understand that the lifetime annotation of the return value can keep the input borrow “alive”.
I haven't fully dived into the variance concept, but this SO answer does summarize it a bit (with a useful analogy) - in short - to that the input argument should outlive the return value. (Of course, I can't really confirm that those takeaways are correct, it does seem to make sense. )
... Combine these two points and the behavior of a function fn foo(&'a T) -> &'a T makes sense: the value passed to the argument must be a subtype of &'a T , so it must live at least as long as 'a . The value returned can only be used as a supertype of &'a T , so it can only be assumed to live as long as 'a . Together we get that the argument outlives the return value.
I simplified my examples, but actually the PhantomData one originated from source code resembling the IterMut<'a>: thanks for the additional reference.
One good way to notice this would also be to replace the implementation of get_mut_value with something like panic!() . The borrow-checking at the call site won’t change from this, at all.
One more question: while I tend to agree to what you're saying (as I understood it the same way), it seems to contradict more or less what is written here in the book:
Lifetime annotations don’t change how long any of the references live. [...]
Because now we seem to conclude that because of this lifetime annotation for the return value, it actually changes how long the array reference lives, further along in the program.
This is a subtle difference indeed, but no, the book is right. It's usually said as "lifetimes are descriptive, not prescriptive" - that is, they describe the relations that already exist in the code, but not change them. What you are referring to is the fact that these descriptions can deviate from the actual behavior (which usually leads to contradiction and compile error), but if the code compiles, then lifetimes doesn't affect the result.
I think the book is rather confusingly written here. The start of the page introduces the following notion:
One detail we didn’t discuss in the “References and Borrowing” section in Chapter 4 is that every reference in Rust has a lifetime, which is the scope for which that reference is valid.
The notion of “valid” here is generally referring to previously mentioned principles, e.g. in a previous chapter already it came up in
Let’s recap what we’ve discussed about references:
At any given time, you can have either one mutable reference or any number of immutable references.
References must always be valid
In the next sentence(s) then,
Most of the time, lifetimes are implicit and inferred, just like most of the time, types are inferred. We are only required to annotate types when multiple types are possible. In a similar way, we must annotate lifetimes when the lifetimes of references could be related in a few different ways.
it sound like “lifetimes” refers directly to the (often-inferred) kind of information that the borrow checker can determine and understand.
In the section after, this is seemingly further confirmed
This section does particularly strongly mix up different notions of “lifetime” under the same name, since the image depicts lexical scope. It also uses the term of “live(s)” (lives longer; does not live as long; … etc.)
The term “live(s)” first came up in the section before that, which I guess we’d also have to look at then. That section contains some first explanations of the compiler error for this same code example; which used “live” in it’s phrasing of “x does not live long enough”
The error message says that the variable x “does not live long enough.” The reason is that x will be out of scope when the inner scope ends on line 7. But r is still valid for the outer scope; because its scope is larger, we say that it “lives longer.” If Rust allowed this code to work, r would be referencing memory that was deallocated when x went out of scope, and anything we tried to do with r wouldn’t work correctly. So, how does Rust determine that this code is invalid? It uses a borrow checker.
This section in the book also talks about how the reference r“is still valid” for the outer scope (which seems somewhat different than the notion of “valid” used elsewhere) and instead calls the entire code “invalid”. E.g. the previously quoted subsequent section puts it as follows in its introduction and conclusion:
The Rust compiler has a borrow checker that compares scopes to determine whether all borrows are valid.
[…]
Now that you know where the lifetimes of references are and how Rust analyzes lifetimes to ensure that references will always be valid, let’s explore generic lifetimes in function parameters and return values.
I wasn’t even aiming to complain about these different section though, just provide the context to understand where and how “lifetime” and “live(s)” comes up, so we can try to make sense of this statement:
Lifetime annotations don’t change how long any of the references live. Rather, they describe the relationships of the lifetimes of multiple references to each other without affecting the lifetimes. […]
I find “live(s)” still a rather unclear term being used here. I think it’s least confusing in statements such as “The program is rejected because […] [t]he subject of the reference doesn’t live as long as the reference.” but I’m not even sure whether or not “how long X lives” is meant to be distinct from “lifetime of X” in this page of the book, or perhaps that’s even inconsistently only sometimes the case
I can imagine multiple valid facts that this statement could be referring to, anyway. One would be that the lifetime annotations “don’t change how long any of the references live” because borrow checking never really changes anything, in can only reject certain programs; which are rejected when the compiler “analyzes lifetimes” and then concludes it is not sure that all references are always valid.
Another reading could be that “how long the reference lives” is a different concept from “lifetime of the reference”, the latter being instead a measure (or rather lower bound) on how long the thing being referenced will live for.
(On that note, it’s probably also unfortunate that the section before “lifetime annotation syntax” already uses a kind of metasyntax for “lifetime” that looks completely identical, using names like 'a and 'b, even though it arguably refers to subtly different things than the actual function of the (always generic) lifetime annotations in function signature (and some other places) in actual Rust syntax & programs.)
In any case, I think this sentence doesn’t really help explain anything on its own.
And there’s a more in-depth coverage of much more relevant insights anyway; the statement here simply seems redundant and not helpful, without further explanation. For example at a later point it states this:
When annotating lifetimes in functions, the annotations go in the function signature, not in the function body. The lifetime annotations become part of the contract of the function, much like the types in the signature. Having function signatures contain the lifetime contract means the analysis the Rust compiler does can be simpler. If there’s a problem with the way a function is annotated or the way it is called, the compiler errors can point to the part of our code and the constraints more precisely. If, instead, the Rust compiler made more inferences about what we intended the relationships of the lifetimes to be, the compiler might only be able to point to a use of our code many steps away from the cause of the problem.
Which is really important, and the way in which the annotation does seem to “affect” things, because it defines a contract, and the compiler then reasons locally based on that contract. Also the fact that the borrow checker really is a checker, and only aims to “ensure that references will always be valid” correctly, not aims to be complete in the sense of understand every single program with this property (references always valid), could probably be highlighted better, because it’s the core reason why it’s completely expected behavior that a bad lifetime signature – an incorrectly defined contract – will make the compiler reject an otherwise “correct” program.
While it's still fresh in your mind… can you try to recall where this idea comes from:
I mean: I can try to imagine the language that would work like that… and fail spectacularly. Not in a sense that such thing is not possible, but in a sense that it wouldn't be feasible: if compiler looks in the body of function to see how things are related to each other… why does such language needs a lifetime markup at all? What may lifetime marks, in such a language, be used for?
Having lifetime markups in a language where compiler looks on the body of functions would imply that authors of such a language are either idiots or sadists… they are asking you to do things that are not needed and can be easily done by a compiler. And I, kinda, hope that tools that I'm using are not written by neither idiots nor sadists thus it was always obvious, for me, that body of function is compared to the lifetimes markup in the function declaration and then said declaration is used for verification of other without without looking on the body: this approach clearly makes sense, it's useful, it have to be like this.
But if not that… then what other justification for the existence of lifetime markup have you had, in your head?
How about we rename lifetimes to loan periods? When I go to a library and borrow a book, the loan period specifies how long I can have it, not how long I actually will have it, or how long the book exists.
I'd hope the Rust compiler might eventually technically fulfill this property of "compiler looks on the body of functions". Though only for it to look at the body of a called function for diagnostic purposes, in case of a borrow checking error at the call site. It would be great if the compiler could throw you that suggestion of "hey, if you change the lifetimes in this function signature like that, then both the function definition and all (visible) call sites will still work".
Even if the compiler could always produce the right function signatures for you to insert (perhaps even automatically with cargo fix) by some global analysis, it can of course still be beneficial to "have" the annotations. It can help you notice if a signature changes even though you never indented it to change. It's still useful for those cases where you do leave the implementation of a function a todo! but already want to be able to borrow check the call sites; or conversely, to borrow check the implementation of a function when you don't have any call sites written down yet but already know what kind of borrow your call site will want to have. And it can make the program just easier to read. (Rust has this similarly slightly "sadistic" property that you're required to write down the function signature of each method in a trait implementation.)
The uses of the return value[1] determines where the initial borrow lifetime remains active. The annotation dictated the relationship between the return value and the initial borrow. The annotation cannot change where the reference itself drops and cannot change where the referent drops either.[2] The borrow checker gives code a pass or fail result; it does not change the semantics of the code.
That being out of the way, I will also say: The book's presentation of borrow checking isn't very good and needs a rewrite, IMNSHO. @steffahn covered some of the reasons, which includes strong conflation of different meanings of the word "lifetime",[3] and straight-up incorrect descriptions of how the borrow checker is implemented.
Chapter 10.3 is particularly bad in this regard, and it's description of how longest works is what I had in mind when I said the book is straight-up incorrect. Attempting to summarize:
The signature with two &'a str arguments does actually mean the lifetimes of the inputs -- as in Rust lifetimes[4] -- must be the same. Variance and automatic reborrowing can make it seem like you can pass in two distinct lifetimes that outlive 'a, but that is not how things actually work.
There is no "replacing of the lifetime with the smaller scope" going on during borrow checking. The borrow checker does not compute some concrete Rust lifetime based on what you pass in at the call site, and then assign that lifetime to the output. Instead it works pretty much exactly opposite of what the book describes: uses of the return value determine the lifetime -- where the referents of the arguments must remain borrowed.
Most of their material about "scopes", trying to visualize Rust lifetimes as lexical blocks, and so on is inaccurate or misleading. Think of a value going out of scope as a use of that value which is incompatible with being borrowed. Other uses which are incompatible with being borrowed include being moved or having a &mut _ taken. That's what their "example that shows that the lifetime of the reference in result must be the smaller lifetime of the two arguments" is actually demonstrating.[5]Here's a modification that demonstrates the &mut case instead.
)
