Blog post series: After NLL -- what's next for borrowing and lifetimes?


#41

It may indeed be typical, but it’s also not uncommon to have multiple impl blocks in the same file, and I’ve also seen impl blocks spread across files (admittedly, the latter seems fairly rare IME). But, I’d be willing to give that feature a shot and see how I feel about it after some experience.

The problem is it doesn’t address the same issue in traits, where the problem is more acute. And if traits end up with a solution, it’d be highly desirable that it share the solution with structs.


#42

The current PaintCtx in druid suffers massively from this problem, and we’ve been working around it by creating all resources before taking a mutable borrow of the render target. The current design makes it very difficult to interleave creating new resources with painting.

Very likely we will go with the view struct approach, but we are considering alternatives. I’m not feeling like it really needs language changes, but it’s definitely something that demands more work to reason through.

People who are interested in the specifics are invited to the Zulip thread about druid future plans, but probably the fate of PaintCtx should become an issue.


#43

I ran into it a few times, but I always just solved it with the factoring approach. IMO, the result is cleaner anyway. I haven’t run into the more complicated cases where splitting doesn’t work, but I’m not sure whether it is worth complicating the mental model of the language to handle those cases. The language will never be handle to handle 100% of cases cleanly anyway without going to full dependent types.


#44

Do you also give a big vote in favor of increasing the complexity of the compiler/language? Language design must balance all the constraints at the same time.

Niko:

In my previous post on the status of NLL, I promised to talk about “What is next?” for ownership and borrowing in Rust.

Perhaps it’s the right moment to pause the increase in borrowing refinement and rules, and work on improving other parts of core Rust language.


#45

As a Rust beginner, I don’t know anything about the compiler, so I’m not in a position to judge how much additional complexity this would add to it. Niko asked how often people ran into this issue, so I answered.


#46

(Didn’t have time to read the intervening conversation, so sorry if this has already been mentioned.)

Maybe this is just my bias, but the solutions described in this post seem like they’re really trying to reinvent structure slicing from the Legion programming system:

Basically, you can different privileges on different fields:

struct s { ... }

task f(r : region(s))
where reads writes(r.x), reads(r.y)
  ...
end

Above, field x is read-write (like a mutable borrow) and y is read-only (like an immutable borrow).

Don’t know if this is really applicable to Rust or not, but it might provide some food for thought.


#47

As a datapoint from a user perspective I can definitely testify that I encounter this issue very commonly. Most of the time I end up refactoring the code to inline the problematic method in the caller to work around the issue but it’s definitely far from perfect and quite frustrating. I guess it’s mostly because I dislike long functions and prefer to factor code in multiple short methods even if they are “single use” and only called from a single location. I find that it makes it easier to think about the code and enforce invariants at the method’s interfaces.

That being said the proposed annotations seem frankly cumbersome to me. In function signatures we can already specify traits, lifetimes, types mutability and now we’d specify which member are accessed? And should you forget to do it properly it’d default to borrowing all the fields (like currently) so if you encounter this issue with third-party code you’d have to pretty-please ask that they fix their signature. That reminds me a lot of throw specifiers in C++ (in that they’re very easy to code wrong and default to the broadest case) and that’s definitely not something that’s generally considered a good feature of the language AFAIK.

Personally I’d much prefer the “magic” proposed by kornel above where the compiler would automatically figure out what method uses what but limit that to “private” interfaces to sidestep the semver issues. Actually I think I’d be fine if it was even more restrictive than that: only allow these “partial borrows” for self calls, that is a class method calling an other method from the same class (including trait methods implemented by that class) but not any call from the outside. I think that would solve most of my issues without adding more syntax. It does add some magic but it’s still fairly easy to reason about IMO.

Of course if it ends up adding a lot of complexity to the compiler and especially if it increases the compilation times significantly it might not be a great move as far as usability is concerned.


#48

Regarding the idea of adding syntax to specify “borrow” info publicly to type signatures I really wonder if that’s a good idea from a usability perspective. It means that we’re effectively exposing potentially private details of our implementation in the interface. So if I want to be add a signature to a method to be able to call it internally with a “partial borrow” but I don’t want to commit to that on the public interface because I know I might want to change the implementation later how would I do that? Would we have even more syntax to specify if you have public or private annotations?

What if I want to specify something like "I don’t know exactly what this method may end up borrowing but I know that it will never borrow x"? That seems like a more reasonable public commitment but that’s even more syntax.

I really don’t think I want to have these annotations (be it explicit or inferred by the compiler) cross through the public interface. And when I actually want to provide a public interface that works around these issues then adding the extra work of adding sub-types or a “view” object is probably warranted and less of a usability issue IMO.

So basically I’m back to the point of my previous post, having compiler magic to infer who borrows what but only for “private” calls seems like a decent compromise to me.


#49

In the borrowing regions proposal you can define either public or private region in the struct definition (pub region vs region), as a consequence you will not be able to use private regions as part of the public method, and thus private regions will not be exposed in the public API.


#50

Ah yes, that makes for much nicer looking public interfaces but at the cost of an even higher cognitive load for the implementer. For public regions you better get it right from the start too since you’re committing to it publicly.

Private regions would have the advantage of reducing the amount of noise if you have a subset of members that are often borrowed together but I still think that for that use case compiler magic would result in much better ergonomics.

Also in this proposal, unless I’m missing something, you wouldn’t be able to define overlapping regions (otherwise you could have conflicts between two different regions) so that might prove rather limiting if method_a uses variables x and y while method_b uses variables x and z. You either need to have a single region for x, y and z for both methods (but then you actually “overborrow”) or three different regions r_x, r_y and r_z and each method uses two of them. That works but could mean that you end up with many regions in complicated borrow scenarios.

I often hear non-Rust users complain that the language looks daunting with its use of the borrow checker, adding even more complexity on top might be overdoing it IMO. The great thing about NLL is that while it’s a bit more complicated in theory, in practice it means that more code becomes valid at no additional cost to the user. No additional typing, no new rules to remember, it basically just works thanks to the compiler being more clever than it used to be and recognizing a broader range of valid, safe code.


#51

I ran into this problem in a big way early in our Rust project and resolved it using free functions with lots of parameters. Since then I have continued to run into it once in a while but have never subsequently felt that it was a big issue (compared to, say, async/await or the non-scalability of RLS). I’m skeptical that any increase in language complexity to address this problem would be worthwhile. Explicit syntax would be hard for new users to discover; implicit inference of struct slices risks the confusion when such inference inevitably reaches its limits. Maybe some simple transformation that works for borrows into closures would be OK.

(I’m a bit worried that when Really Smart People study problems in programming languages there’s a cognitive bias towards “this is a problem we can solve, therefore we should solve it”.)


#52

There’s something really simple and elegant in C having all code in functions at a global namespace. You can say “the function you want is X” and there can be no question or confusion about how to call it or where it can be called. “Find the name of function X and read its documentation” is pretty much the only thing you need to do to understand anything. (Aside from all that pesky UB nonsense.)


#53

I run into this a lot. When I’m coding on my SAT solver these days, it feels like the only thing I’m doing is shuffling the workarounds required due to this. I’d greatly appreciate a language solution to this. In fact I just started writing a Pre-RFC guided by the main issues I had with implementing my SAT solver, before realizing that there is an active discussion about this again.

If it’s not a complete duplication of efforts I would continue with that (I’ll post a first draft soon) and would also be up for implementing that or a different proposal. (I haven’t worked on rustc yet, but started looking into it, as my SAT solver is unstable rust anyway and I feel like I’m blocked on this, so having some solution has high priority for me.)


#54

There’s something really simple and elegant in C having all code in functions at a global namespace. You can say “the function you want is X” and there can be no question or confusion about how to call it or where it can be called. “Find the name of function X and read its documentation” is pretty much the only thing you need to do to understand anything. (Aside from all that pesky UB nonsense.)

I’m not sure how this insight is relevant to the subject at hand but I disagree about the “elegant” part. It’s simple and barebones but lacks a lot of expressiveness. Take a C prototype like this for instance:

some_type *my_function(some_other_type *param)

What does this prototype tell you? Well not enough to use the function safely, that’s for sure. Can the parameter be NULL? Does the function take ownership of param or can I keep using it afterwards? What about the return value, do I own it? Do I need to free it? If so how? Does it borrow or own param at all? Can the function return NULL in case of an error? Or maybe an ERRNO code masquerading as a pointer somehow?

In Rust all these questions are answered by the function signature without having to find the doc (if it even exists and is accurate) or having to look at the implementation and you don’t even have to pay too much attention because if you use it wrong the compiler will yell at you:

fn my_function<'a>(param: Option<&'a SomeOtherType>) -> Result<SomeType<'a>, ()>

Sure in C you don’t have to bother with the borrow checker, partial borrows or NLL but I don’t think it makes the language more or less “elegant”, it just makes it simpler and less safe.

Also namespacing is a completely orthogonal issue IMO.


#55

While I like the idea of automatic partial borrows across private methods, I don’t think that would be enough. I’ve often written code using one of the mentioned workarounds where the functions in question were public.

I thus think extra syntax has to be part of the solution and in that case I’d suggest requiring annotations for the beginning and optionally adding inference for private scoped functions later. I fear that extending the borrow checker to work across functions without considering how it could be encoded in function signatures might make it difficult to add such syntax later.


#56

Could you (or other participants) share specific examples of encountering this issue across public interfaces? That might be because of my personal coding style but while I do encounter “private” partial-borrow issues regularly I can’t think of a single instance where that turned out to be a problem with external code. I’d be very curious to see what that looks like in practice so that I can better understand the problem.


#57

I often want to have structs that encapsulate a collection of items and some associated data. For example a graph containing vertices and edges. It would be nice if I could iterate over the collection of vertices and modify the associated data, the edges, from outside the struct. For example because I’m writing code that solves a specific graph problem, so it shouldn’t go into the impl of my more general graph struct. But I also want to have functions that remove a vertex and all associated edges, so I do want to have a single struct containing both vertices and edges.

It gets even more tricky if I have two collections that are related encapsulated in a struct (for example the two vertex partitions of a bipartite graph) and want to iterate over one while modifying the other in both directions (not at the same time though).

I often work on problems that have parts that are somewhat similar to these.

I don’t really have much existing code that is useful as a small self-contained example for this, as I had to rework my preferred designs into something that I can actually implement. I am quite unhappy though with the way I had to architect my SAT solver to work around this and am in the progress of figuring out how I would want to refactor it assuming I had whatever language features needed. I might come up with more specific examples in the progress of doing so.


#58

I think you’re misunderstanding me. My point was not that borrowing is worse than pointers (or that C’s type system is anything worth emulating at all), but that specifying borrow behavior using plain functions which pass their arguments uniformly is not something to be avoided. Not every function needs to be a method, and when someone hits a problem where borrowing self locks them out of some data access patterns, then the existing solution of writing a different function is not an inelegant one, because it is simple.

With that said, I’m not actually opposed to the existence of some finer-grained borrow-specifying mechanism. I just don’t think it should be the default way of doing things, and thus don’t give much weight to making the syntax for it as unobtrusive as possible. (In particular, I don’t think it should be invisible!)


#59

Threading a bunch of references through multiple functions, just because one function deep in the call stack needs them is not something I consider elegant, even though I agree it is simple. It is a workaround I have to use often and would prefer not to. A view struct only helps if the set of references is the same for a group of functions.

There is also a performance issue, because the compiler can’t know all the references are just offsets of the same base so all have to be stored in registers or on the stack or in a separate view struct. I intend to do some benchmarks for this.


#60

I wonder what it would take to let the borrow checker follow along combinator-chained futures past yield points. If I remember correctly, part of the impetus for async/await was the borrow checker’s inability to do so. And I personally like to do chains of combinatory methods :slight_smile: