Workaround for: cannot borrow `*self` as mutable more than once at a time


#6

Thanks. To be clear, the error does not go away if the reference to the internal SizeClass returned is immutable, that is, this snippet produces the same error. Even if the first method that is declared &mut self returns an immutable reference to a location inside the object, further attempts at borrowing the object mutably will fail.

I understand the argument that an internal reference could be modified in a way that invalidates it in theory (for other examples such Vec! I’ve seen in this forum), but my question was about my example, not in theory. How can an embedded, fixed-sized array consisting of structs with one primitive type field be modified in a way that would invalidate it when performing further mutable borrows of the containing struct?

Regarding @Michael-F-Bryan’s proposed work-around, I’m struggling to understand how this would work for my case because in a memory allocator, implementing alloc() involves accesses to multiple SizeClasses. For instance, if you allocate n bytes, you may need to get access to the size classes for n/2 bytes and possibly n/4, etc. bytes. My example was simplified (in the actual implementation, the size class contains a linked list of blocks to which blocks need to be added or removed from as part of implementing alloc()).
In short, it’s not possible to implement ‘alloc’ in a method of SizeClass - it needs to be implemented in Allocator (which is also the logical place for it to be.)


#7

Immutable or mutable doesn’t matter here - you cannot hold two mutable or a mutable + immutable references at the same time.

The compiler assumes no aliasing. In your _do_allocate(), it may optimize the code with that assumption which means, if you were to violate this assumption, it can generate code that will be incorrect (eg a store will not be noticed).

The compiler cannot reason about code at the level you’re asking. Instead, you need to explain what’s borrowed how using its own rules, which I mentioned earlier in the linked thread. Basically, you need to borrow fields individually so that it can see you don’t have overlap.


#8

Ok - let me see if I understand this.

Since Rust no longer supports immutable struct fields, and apparently since the borrow checker does not perform interprocedural analysis, the compiler acts as if one of the &mut self methods could do something like:

self.classes[0] = SizeClass { sz : 50 }

which would mean that if there were an alias to self.classes[0] (like that returned by find_size_class()) the compiler might incorrectly use values obtained from the old reference.

Does it seem as though Rust is baking a subtle limitation on the compiler’s ability to perform alias analysis into the language? In my example, I certainly do not intend to change the ‘classes’ array ever after construction and I would like to mark it immutable/final/const if there were a way to do so.


#9

Well, here is the tricky part: as far as I understand it, the fact that the borrow checker does not perform interprocedural analysis is actually more than an implementation limitation. It is a conscious design choice. And the rationale for that choice is that if you did things differently, you could introduce backwards-incompatible changes in your code just by modifying a method’s implementation, without touching its interface (or otherwise violating it in the code). As a result, every borrow which you carry out inside of the implementation would effectively become an invisible part of its interface, a form of abstraction leakage which is generally considered undesirable in Rust.

This is the same reason for which Rust does not allow function parameters and result types to be inferred. If it did, then changing something deep into the internals of a library could break its clients in the kind of subtle ways that heavy users of C++ templates and dynamically typed languages have learned to fear, without visibly touching any public API. As a result, evil gigantic compiler backtraces would spread across the Rust world, making humanity miserable again, and maintaining a healthy library ecosystem without freezing implementations would become much harder.

There are periodical discussions about introducing new extensions to the Rust interface design vocabulary, which would allow a developer to make it explicit that a certain object method only borrows part of the object, and that another method operates on a disjoint part of the object. If these extensions are ever introduced to the language, then Rust will gain more global borrow checking. In meantime, the easiest answer is usually, as @vitalyd explained, to avoid encapsulating your implementation details too much, instead directly manipulating concrete types and struct members.


#10

I’m with @HadrienG here.

So here’s another (potential) option for you - return a clone or copy of SizeClass, rather than a reference. If this is truly a value object, then that should be feasible and you don’t need to worry about borrows in this case.


#11

Ok, the language design reasons aside… what’s the work-around/Rust pattern.

Vitaly discusses using a function instead of a method here - however, this leads to the same issue for me, as shown in this playground. This is so, because, presumably in the code below taking the mutable reference to &mut self.classes is considered a mutable reference to self:

    fn alloc(&mut self, sz: usize) -> Result<bool, AllocErr> {
        let clazz = find_size_class(&mut self.classes, sz)?;  // step 1: find size class
        self._do_allocate(clazz, sz);  // step 2: allocate
        Ok(true)
    }

I suppose I don’t understand proposal #2.

I think I’ll next try the index-based approach which @mortoray referred to as “hard to read and maintain”.

The last approach, if I understand @vitalyd correctly, would be to wrap the ‘classes’ array in a private struct … but I don’t understand how this will go around the problem that as soon as I take a mutable reference to it, the entire struct will become unusable for the remainder of the function because of the risk of aliasing.


#12

The idea would be to make _do_allocate an associated function, which means it no longer takes &mut self but rather explicit arguments of state that you need. It’s hard to say more for your case because I don’t know what _do_allocate()` would do with other state/fields of the allocator.


#13

The actual implementation is not truly a value object - it contains a LinkedList head, so clone is not an option since the head may need to be updated. What are my options if, say classes[i].sz is immutable/constant but classes[i].free must be mutable? Also, FWIW, the linked list’s head is not being reallocated, it’s simply updated in place.


#14

You can use interior mutability:

struct SizeClass { sz: usize, free: RefCell<YourType> }

Then you can have a &SizeClass borrow (and just a &self borrow of the allocator), and mutate free via the RefCell.


#15

_do_allocate needs access to potentially all elements of classes[] to add/remove things from the linked lists embedded in these.


#16

Re: the RefCell proposal. I will admit that I haven’t studied Rust deep enough to know what that is, but I want to point out that my code is inside the memory allocator I want Rust itself to use. So if RefCell (via RefCell::new() or similar) requires memory allocation, it’s not an option for me because, well, I’m implementing the memory allocator Rust itself uses.


#17

It does not.


#18

So this will be a bit problematic because separate slots in an array are not considered disjoint by borrowck. Typically people will use functions like split_at_mut() to split the allocation and get disjoint mutable borrows that you can hold at the same time. You can also use unsafe code yourself and ensure that you don’t violate aliasing requirements.

Another option is to wrap SizeClass in an Option and take the size class out of the array momentarily, mutate it, and then put it back. This will give you temporary ownership over the size class within the function. The problem, of course, is you may forget to put it back into the array, leading to bugs.


#19

The method in question (find_size_class) is a private helper method local to this class, so considerations related to abstraction leakage may not apply.


#20

This brings us to another popular discussion topic in Rust RFCs: should internal code be allowed to ignore some of Rust’s opinionated rules? Examples of previous discussions include:

  • Can we have inferred types on internal methods?
  • Can we put an entire internal module in an unsafe block?
  • Can we locally change the meaning of array indexing to remove bounds checking?
  • Can we have C++-like duck-typed generics in internal code?
  • Can we allow for shared mutable aliasing in internal code?

So far, however, the consensus has been that having simple and consistent language rules is more important than being allowed to break the rules in internal code. From daily experience with C++, where many language rules are easy to evade, I tend to agree with this assessment.


#21

I think it’s perfectly reasonable to reach for unsafe code if that’s what it takes to achieve the desired functionality (and/or performance). Just need to be careful with that code and not snooze since the compiler won’t help detect violations. In @godmar’s case, that might be a valid approach since the root issue is not being able to convince the compiler that accesses are valid. In particular, since the functions/methods in question are private, the unsafety could be fully self-contained in the impl (or parts thereof, rather). I’d explore safe impl choices first, but if nothing palatable comes of it, I’d go with unsafe (and a quick prayer! j/k :slight_smile:).


#22

Oh, I was not intending to mean that using unsafe would be unnecessary here. My point of view on this matter is currently neutral: I do not understand @godmar’s intent well enough to tell if reaching for unsafe is called for or not, and would personally need more code before forming a firm opinion on this matter.

All I was trying to do is to explain a bit why the Rust language has this peculiar design which often causes much trouble to newcomers from other languages. It was meant as an encouraging thought, in a “we mean no harm, and are not trying to sell you a broken product, everything is working as intended and if you give it a bit of time I hope you will find Rust to become a friendly fellow” kind of way.


#23

I intended that post for @godmar, it wasn’t a rebuttal to your points :slight_smile:

I think the issue there is going to bottom out in borrowck not understanding disjointness of array slots. If it’s the case that conflicting references need to be alive at the same time, it won’t borrow check.

We’ve discussed some ways to deal with that but given this is an allocator, it might not be prudent to pay perf penalties, even if ever so slight, and instead just take the unsafe plunge. But maybe not - I agree that it depends on the exact code.


#24

To benefit from Rust’s promise of efficient and typesafe code, I’d like to keep the amount of unsafe code to a minimum. In addition, I do not understand how the compiler’s understanding of aliases would even interact with unsafe code. In unsafe blocks, does the compiler assume references may be aliased (as in C code) and issue the necessary memory references?

I will admit that I don’t understand why it is apparently so difficult to maintain references to internal fields that do not change (and why the ability to tell the compiler that a field doesn’t change was removed, but I haven’t picked apart the corresponding thread I found online and linked to earlier in this thread). Naively, it seems that the immutability of an object is separate from the immutability of its components (that’s at least how it is in other languages). In Rust, the two seem to be interwoven for reasons that aren’t quite clear. Maybe consider adding a chapter to the 3rd edition of the book.


#25

Here is a modification to your last playground that uses raw ptrs (no unsafe code yet because there’re no derefs of the ptr).

It’s not really unsafe per se but rather raw ptr usage that will allow escaping the sandbox; deref’ing them will require unsafe blocks but that’s separate. A raw ptr isn’t tracked by borrowck. The aliasing and lifetime aspects still need to be enforced, except you do it manually now - compiler assumes you got it right but isn’t able to check it.

It’s not difficult to maintain references to fields as long as they’re disjoint, as we’ve discussed. Moreover, disjointness is important only if you want to hold a unique/mutable reference.

The difficulty is more to do with compiler not understanding disjointness beyond fields.