First post in a series of thoughts about "tricky problems with the borrow checker, even after non-lexical lifetimes (NLL)":
http://smallcultfollowing.com/babysteps/blog/2018/11/01/after-nll-interprocedural-conflicts/
Follows up on my NLL status update post from yesterday.
As part of my (0.0.z) type_level_values crate,I wrote an example where I emulate the views idea.
In that example ,one can "split" a mutable reference so that one only has access to certain fields.
These are some of the tradeoffs I had to make to emulate views:
I didn't even try to implement views in traits,so I would be interested in what solutions other people come up with,either using const-generics or a library which provides the necessary derives.
For member functions that borrow part of self, I wonder if syntax like the following would be accepted:
fn signal_event(&mut self: Self{counter, listener}) {
self.counter += 1;
self.listener.send(()).unwrap();
}
pub fn matcher_with_entities<'s, Q>(
&'s self,
) -> impl Iterator<Item = (Entity, <<Q as Query<'s>>::Iter as Iterator>::Item)> + 's
where
Q: Query<'s> + Matcher,
Q::Borrow: RegisterBorrow,
{
self.borrow_and_validate::<Q::Borrow>();
// We need to explicitly tell rust how long we reborrow `self`, or the borrow checker
// gets confused.
let s: &'s Self = self;
let iter = self
.storages
.iter()
.enumerate()
.filter(|&(_, storage)| Q::is_match(storage))
.flat_map(move |(id, storage)| {
let query = unsafe { Q::query(storage) };
let entities = s.entities_storage(id as StorageId);
Iterator::zip(entities, query)
});
BorrowIter { world: self, iter }
}
I had to explicitly reborrow self with a lifetime here let s: &'s Self = self; and then move the borrow into the closure of flat_map.
Without doing that I would get an error message that would say:
error[E0597]: `self` does not live long enough
--> src/lib.rs:330:32
|
328 | .flat_map(|(id, storage)| {
| --------------- value captured here
329 | let query = unsafe { Q::query(storage) };
330 | let entities = self.entities_storage(id as StorageId);
| ^^^^ borrowed value does not live long enough
...
334 | }
| - `self` dropped here while still borrowed
|
note: borrowed value must be valid for the lifetime 's as defined on the method body at 313:34...
--> src/lib.rs:313:34
|
313 | pub fn matcher_with_entities<'s, Q>(
| ^^
which was super confusing because I explicitly annotated self with &'s self. I only solved it because someone on discord told me that the closure reborrows self, and then after I desugared the closure in my head I knew exactly what was going on.
Here's a rough sketch of how the views could work:
// Defining a view for a type essentially defines a type alias
// that we can use in references, but the compiler knows
// that only certain parts of the actual type are visible.
view EventSignal for MyStruct {
mut counter,
listener,
}
// We can implement methods for the views.
// They are inherited by the full non-view type.
impl EventSignal {
fn signal_event(&mut self) {
self.counter += 1;
self.listener.send(()).unwrap();
}
}
impl MyStruct {
fn check_widgets(&mut self) {
for widget in &self.widgets {
if widget.check() {
// All view methods are available
self.signal_event();
}
}
}
}
fn foo(bar: MyStruct) {
// We can create references to views
let my_view: &mut EventSignal = &mut bar;
// If you create multiple, the compiler can see
// if they overlap in an inconsistent way.
}
// Functions can use the views just like the original type,
// but only have access to the fields of the view.
fn some_fn(the_view: &mut EventSignal) { }
There might additionally need to be a solution for defining which Views are overlapping in what way, so that the compiler can automatically verify for you that you don't accidentally break semver.
Did your original code not work if you had made the closure move as well? That should've copied the reference to self over, no different than you doing it manually via a local binding.
I fear that, except for some low hanging fruits, future changes to NLL will probably require a lot of work while delivering a decreasingly amount of value.
No one else wants to implement the views feature as a library before creating an RFC (I implemented a prototype as an example) ? This will especially be possible emulate with a library once const-generics come.
What I'm most interested to hear about is how often this sort of problem arises for folks (particularly relative to future problems I plan to list) -- in the past, when I've discussed this with people, they often feel it doesn't come up a lot. But ever since I started paying attention, I realize that I think I hit variants of this problem a lot...
I realized after posting this blog post, for example, that my old post about rooting an Rc handle as in some sense wrestling with the same problem.
That is, there is another workaround -- similar to views -- where you put immutable data in an Rc, so that you can clone it locally and then get on with your life.
I'm sure I've come across this problem in the past and found relatively easy ways to get around it, but the ways I got around it were only "relatively easy" because I've been coding in Rust for quite some time now and have a (somewhat) decent understanding of the language. That's not the case for beginners. My point is that I think a feature that solves "interprocedural conflicts" in the borrow checker would be very welcome for Rust's ergonomics and would go a long way to help beginners adopt the language, regardless of the responses you receive on this forum, which are likely to come from folks already relatively familiar with Rust. Basically, I'm trying to point out that you have a "selection bias" problem if you're only relying on this forum for feedback (which you are probably are not). I definitely don't want to imply that you aren't already aware of this issue and incorporating it into your thought process. I just thought it would be helpful to make it explicit.
Heh, yes. I'm definitely aware of that. But it's useful information, even if it must be weighted appropriately. Anyway, I hope -- after a few more posts -- to open up a kind of survey, so that will let us get more quantitative when comparing different patterns.
I agree a lot with what you're saying here. Thinking back, I'm pretty sure I run into this constantly, but it's become so second nature to work around it now that I often work around it before it even comes up. I see it a lot in student code and in the code of coworkers who are new to Rust though. For myself, I usually now only run into it for the really bad cases, and then one of the workarounds you gave usually works after a bit of re-engineering. The other trick I sometimes end up using is mem::replace(&mut self.foo, Default::default()), which is super ugly, but also does the job in nearly all situations. It's very very hacky though.
EDIT: specifically, I think a decent number of these are that kind of workaround. Some are old, and could probably be worked around with more code reorg though.
Thanks @jonhoo. By the way, life has gotten in the way, but I'm working on addressing your PR shortly!
This issue comes up a lot! 
I've seen countless posts on this forum alone where the root issue was exactly the interprocedural conflict described in the blog. This is a particularly frustrating issue for newcomers because they've, very likely, never had to make these considerations in whatever language(s) they already know. The "state splitting" technique (i.e. arranging fields into different structs to fit the borrowing usage) is frequently touted as a "work around".
I think once people get over the initial jarring effect of this, it mostly works out OK when using concrete structs and they even like it because the code is arguably clearer. But this gets more difficult once you start modeling abstractions via traits, where (a) one feels more pressure to get the design right and (b) there's no innate disjointness to lean on, like with structs.
I actually think this added borrowing dimension in the API design is what holds a lot of people back from publishing their crates as 1.0 - they have this conscious (or maybe even subconscious) fear that their API isn't quite right, and I suspect the interprocedural aspect doesn't help there .
Thanks a lot for writing this up. This issue definitely comes up a lot, having a deep influence on API designs using Rust language!
I've been getting quite used to this "restriction" now, Whenever i need potential "getter"-style access to one of the fields, i just access the field directly. This does need me to organize the fields in a clean, understandable way, which is not a bad thing itself.
The "view struct" pattern is new to me, I think i'll give it a try as an experiment.
Overall i'm not feeling bothered with the current state. At the same time i do have observed beginners getting tripped over this a lot - enough times until they found this "restriction" by themselves.
Maybe, just maybe, that Rust can gain the ability to describe what self is actually borrowing? Only then can it provide Java-like getters. I mean, some fictional syntax like(though the syntax may be ugly):
struct S {
a: A
b: B
}
impl S {
pub fn a(&self{a, ..}) -> &A { &self.a }
pub fn set_a(&mut self{a, ..}, a: A) { self.a = a; }
pub fn b(&self{b, ..}) -> &B { &self.b }
pub fn set_b(&mut self{b, ..}, b: B) { self.b = b; }
}One thing to keep in mind is the complexity of the mental model of borrowing. Before NLL it was simpler to reason about, which really helped in understanding borrowing errors even if they were a pain to work around sometimes. The "smarter" the borrow checker gets, the harder it gets to reason about borrowing errors. Very precise error messages will be helpful with that.
Making borrowing harder to understand may just add a new barrier for beginners that they only reach later when their code gets more complex. So encouraging simpler design patterns at an earlier stage may be a better strategy in the long run.
Making borrowing harder to understand may just add a new barrier for beginners that they only reach later when their code gets more complex. So encouraging simpler design patterns at an earlier stage may be a better strategy in the long run.
I think this is a reasonable and thoughtful sentiment, but as a counterpoint, if users are only hitting borrow errors in sufficiently complex code, then that implies that borrow errors are fundamentally subtle things that are part of complex systems, and we should defer those errors to the times when people choose to interact with subtle and complex systems.
Allow me to make an analogy; pretend that we all agree that violence is a bad thing. One might be tempted to propose that we should extend that badness qualifier to all things which can reasonably be expected to enable violence, such as the throwing of knives. However, the actual goal here should be to get people to understand what violence is, not in terms of whether throwing knives leads to it, but in terms of the harm that it induces in the world. People should be able to analyze a situation and see the violence in it, not proscribe things as violent because they are on a simplified list of violent things.
Rust programmers should only get errors in code that is wrong... or rather, not demonstrably correct. I don't think the Rust compiler's job is to fundamentally misunderstand borrowing semantics in the name of simplicity, just to teach beginners a simple-but-incorrect model of what is reasonable to borrow and when. Though I think you could recover the same benefits by having some kind of "strict mode" which could be opted into for a more complex borrowing model.
@nikomatsakis this is also very similar to finding a good way to express interior vs exterior mutability of collections in function parameters.
While it is a solved problem for the most common usecase Vec -> &mut [] and String -> &mut str, I have found myself wanting a way to express interior mutability for Sets, Maps, etc. i.e. Any situation where people use a non-aliased Cell or RefCell.
I like borrow regions idea, which essentially got reinvented in this thread as well. I think it's a quite natural extension which will allow us to take finer grained borrows without any tedious refactorings like those listed in the post. Additionally it works nicely with traits as well and can be seen as an alternative to "fields in trait" approach.
And talking about future after NLL, can we do something about self-referential structs? At the very least some sort of 'self lifetime would've been nice.
I feel like this would be a combination of a lightweight syntax for creating a view type, along with arbitrary self types.
I think you'd need to include in this syntax specifiers for how you are borrowing the different parts of the struct; for example, if a method needed to borrow widgets immutably as well, you'd need to be able to write something like:
fn foo(&mut self: Self {&widgets, &mut counter, &mut listener}) {
// ...
}
The if you wanted to have a view type that could be shared between methods and not just defined for this one method, you could just use an alias:
type SomeView = MyStruct {&widgets, &mut counter, &mut listener};
type SomeOtherView = MyStruct {&mut widgets};
type SomeThirdView = MyStruct {&mut counter, &mut listener};
fn foo(&mut self: SomeView) { }
Of course, this syntax may not be possible or desirable, other colors of the shed welcome, but it seems like a lightweight syntax for defining one of these view structs, along with arbitrary self types that could automatically build a view struct from a struct, could do the trick here.