Workaround for: cannot borrow `*self` as mutable more than once at a time


#1

I am new to Rust. The pattern I am trying to implement is to invoke multiple methods on an object, what in other languages would typically be expressed as:

   let obj = ....
   obj.call1();
   obj.call2();
    ...

whereby call1 and call2 both take a &mut self as their first argument.

However, I receive the error:

error[E0499]: cannot borrow `*self` as mutable more than once at a time

I don’t understand whether this error indicates a true problem in my code or a limitation of the compiler. Most online sources related to this error point to the unimplemented NLL feature, however, none I’ve found discusses work-arounds. (This includes the 4 posts in this forum with a similar title.)

Either way, not only do I not understand this message, but I haven’t been able to find a work-around of any kind… so my question is: how do I invoke multiple methods (in sequence) that mutate an object; or conversely, why is it unsafe/illegal to invoke multiple methods that mutably borrow self in sequence?

My code is below, with unessential details elided:

enum AllocErr { Bad }

struct SizeClass { sz: usize }

struct Allocator {
    classes : [SizeClass; 3]
}

impl Allocator {
    // create an allocator with 3 size classes
    pub fn new() -> Allocator {
        return Allocator { 
            classes : [
                SizeClass { sz: 8}, SizeClass { sz: 16}, SizeClass { sz: 32}
            ]
        }
    }

    fn find_size_class(&mut self, sz: usize) -> Result<&mut SizeClass, AllocErr> {
        match self.classes.iter_mut().find(|clazz| { clazz.sz > sz }) {
            Some(clazz) => Ok(clazz),
            None => Err(AllocErr::Bad)
        }
    }

    fn _do_allocate(&mut self, _clazz: &mut SizeClass, _sz: usize) {
    }

    fn alloc(&mut self, sz: usize) -> Result<bool, AllocErr> {
        let clazz = self.find_size_class(sz)?;  // step 1: find size class
        self._do_allocate(clazz, sz);  // step 2: allocate
        Ok(true)
    }
}

fn main() { Allocator::new().alloc(42); }

Any help is appreciated.


#2

These sorts of error messages almost always point to a design issue because you’re doing something not 100% correct, not a limitation of the compiler. Blaming the tools is my first reaction as well, but I’ve found the Rust compiler is almost always correct and it’s detected when I’m doing something wrong (or rather, not 100% right).

I believe the underlying issue is with how you’ve designed the “size class” lookup in that it’ll mutably borrow the allocator for the entire time a size class reference is around. A better way to design your allocator is so each SizeClass acts as a bucket and you allocate things inside your SizeClass. That way you aren’t trying to mutably borrow bits of the allocator more than once at a time.

Here’s an example of how I’d do things.


#3

I can answer this one:

To be clear, the issue here is not with mutating an object multiple times in a sequence. For example, pushing to a Vec multiple times in a row is fine. Where Rust draws the line is when you start mutating an object while holding a reference to its internals.

When you do this, there is a fair chance that the state to which you are holding a reference will be modified, in a way that can invalidate the reference. For example, pushing to a Vec while you hold a reference to it (such as an iterator) can lead to a reallocation, in which case your reference would become dangling. Depending on which language you are using, this could be an unpredictable runtime error (Java, C#) or security-critical undefined behaviour (C, C++). Rust cannot stand for either, so it prevents them by design.

As a consequence, idiomatic Rust code typically avoids holding long-lived references to mutable object internals and favors alternate approaches that use immutable references or independent values more (as in functional languages), or that only keep the references alive as long as necessary for a given transaction. @Michael-F-Bryan’s example shows one way of doing this that may work in your particular case.


#4

Thanks @HadrienG, that’s something I was trying to get at, but you’ve worded it a lot better than me!

If only there was a way to sticky your response. Once you’ve learner the basic mechanics of a language and are moving onto more interesting stuff, these sorts of design considerations become really important. Having an understanding of why Rust does things the way it does and what problems it’s trying to prevent can really help to improve your code (and not just in Rust).


#5

In terms of implementation approaches to avoid these errors, some of them are discussed here.


#6

Thanks. To be clear, the error does not go away if the reference to the internal SizeClass returned is immutable, that is, this snippet produces the same error. Even if the first method that is declared &mut self returns an immutable reference to a location inside the object, further attempts at borrowing the object mutably will fail.

I understand the argument that an internal reference could be modified in a way that invalidates it in theory (for other examples such Vec! I’ve seen in this forum), but my question was about my example, not in theory. How can an embedded, fixed-sized array consisting of structs with one primitive type field be modified in a way that would invalidate it when performing further mutable borrows of the containing struct?

Regarding @Michael-F-Bryan’s proposed work-around, I’m struggling to understand how this would work for my case because in a memory allocator, implementing alloc() involves accesses to multiple SizeClasses. For instance, if you allocate n bytes, you may need to get access to the size classes for n/2 bytes and possibly n/4, etc. bytes. My example was simplified (in the actual implementation, the size class contains a linked list of blocks to which blocks need to be added or removed from as part of implementing alloc()).
In short, it’s not possible to implement ‘alloc’ in a method of SizeClass - it needs to be implemented in Allocator (which is also the logical place for it to be.)


#7

Immutable or mutable doesn’t matter here - you cannot hold two mutable or a mutable + immutable references at the same time.

The compiler assumes no aliasing. In your _do_allocate(), it may optimize the code with that assumption which means, if you were to violate this assumption, it can generate code that will be incorrect (eg a store will not be noticed).

The compiler cannot reason about code at the level you’re asking. Instead, you need to explain what’s borrowed how using its own rules, which I mentioned earlier in the linked thread. Basically, you need to borrow fields individually so that it can see you don’t have overlap.


#8

Ok - let me see if I understand this.

Since Rust no longer supports immutable struct fields, and apparently since the borrow checker does not perform interprocedural analysis, the compiler acts as if one of the &mut self methods could do something like:

self.classes[0] = SizeClass { sz : 50 }

which would mean that if there were an alias to self.classes[0] (like that returned by find_size_class()) the compiler might incorrectly use values obtained from the old reference.

Does it seem as though Rust is baking a subtle limitation on the compiler’s ability to perform alias analysis into the language? In my example, I certainly do not intend to change the ‘classes’ array ever after construction and I would like to mark it immutable/final/const if there were a way to do so.


#9

Well, here is the tricky part: as far as I understand it, the fact that the borrow checker does not perform interprocedural analysis is actually more than an implementation limitation. It is a conscious design choice. And the rationale for that choice is that if you did things differently, you could introduce backwards-incompatible changes in your code just by modifying a method’s implementation, without touching its interface (or otherwise violating it in the code). As a result, every borrow which you carry out inside of the implementation would effectively become an invisible part of its interface, a form of abstraction leakage which is generally considered undesirable in Rust.

This is the same reason for which Rust does not allow function parameters and result types to be inferred. If it did, then changing something deep into the internals of a library could break its clients in the kind of subtle ways that heavy users of C++ templates and dynamically typed languages have learned to fear, without visibly touching any public API. As a result, evil gigantic compiler backtraces would spread across the Rust world, making humanity miserable again, and maintaining a healthy library ecosystem without freezing implementations would become much harder.

There are periodical discussions about introducing new extensions to the Rust interface design vocabulary, which would allow a developer to make it explicit that a certain object method only borrows part of the object, and that another method operates on a disjoint part of the object. If these extensions are ever introduced to the language, then Rust will gain more global borrow checking. In meantime, the easiest answer is usually, as @vitalyd explained, to avoid encapsulating your implementation details too much, instead directly manipulating concrete types and struct members.


#10

I’m with @HadrienG here.

So here’s another (potential) option for you - return a clone or copy of SizeClass, rather than a reference. If this is truly a value object, then that should be feasible and you don’t need to worry about borrows in this case.


#11

Ok, the language design reasons aside… what’s the work-around/Rust pattern.

Vitaly discusses using a function instead of a method here - however, this leads to the same issue for me, as shown in this playground. This is so, because, presumably in the code below taking the mutable reference to &mut self.classes is considered a mutable reference to self:

    fn alloc(&mut self, sz: usize) -> Result<bool, AllocErr> {
        let clazz = find_size_class(&mut self.classes, sz)?;  // step 1: find size class
        self._do_allocate(clazz, sz);  // step 2: allocate
        Ok(true)
    }

I suppose I don’t understand proposal #2.

I think I’ll next try the index-based approach which @mortoray referred to as “hard to read and maintain”.

The last approach, if I understand @vitalyd correctly, would be to wrap the ‘classes’ array in a private struct … but I don’t understand how this will go around the problem that as soon as I take a mutable reference to it, the entire struct will become unusable for the remainder of the function because of the risk of aliasing.


#12

The idea would be to make _do_allocate an associated function, which means it no longer takes &mut self but rather explicit arguments of state that you need. It’s hard to say more for your case because I don’t know what _do_allocate()` would do with other state/fields of the allocator.


#13

The actual implementation is not truly a value object - it contains a LinkedList head, so clone is not an option since the head may need to be updated. What are my options if, say classes[i].sz is immutable/constant but classes[i].free must be mutable? Also, FWIW, the linked list’s head is not being reallocated, it’s simply updated in place.


#14

You can use interior mutability:

struct SizeClass { sz: usize, free: RefCell<YourType> }

Then you can have a &SizeClass borrow (and just a &self borrow of the allocator), and mutate free via the RefCell.


#15

_do_allocate needs access to potentially all elements of classes[] to add/remove things from the linked lists embedded in these.


#16

Re: the RefCell proposal. I will admit that I haven’t studied Rust deep enough to know what that is, but I want to point out that my code is inside the memory allocator I want Rust itself to use. So if RefCell (via RefCell::new() or similar) requires memory allocation, it’s not an option for me because, well, I’m implementing the memory allocator Rust itself uses.


#17

It does not.


#18

So this will be a bit problematic because separate slots in an array are not considered disjoint by borrowck. Typically people will use functions like split_at_mut() to split the allocation and get disjoint mutable borrows that you can hold at the same time. You can also use unsafe code yourself and ensure that you don’t violate aliasing requirements.

Another option is to wrap SizeClass in an Option and take the size class out of the array momentarily, mutate it, and then put it back. This will give you temporary ownership over the size class within the function. The problem, of course, is you may forget to put it back into the array, leading to bugs.


#19

The method in question (find_size_class) is a private helper method local to this class, so considerations related to abstraction leakage may not apply.


#20

This brings us to another popular discussion topic in Rust RFCs: should internal code be allowed to ignore some of Rust’s opinionated rules? Examples of previous discussions include:

  • Can we have inferred types on internal methods?
  • Can we put an entire internal module in an unsafe block?
  • Can we locally change the meaning of array indexing to remove bounds checking?
  • Can we have C++-like duck-typed generics in internal code?
  • Can we allow for shared mutable aliasing in internal code?

So far, however, the consensus has been that having simple and consistent language rules is more important than being allowed to break the rules in internal code. From daily experience with C++, where many language rules are easy to evade, I tend to agree with this assessment.