Solving lifetime and multiple borrows concerns with custom iter_mut

I'm using a custom tree structure using vectors to store data and indices for the edges instead of pointers. Making an iterator with immutable references was fine, but I had to work around lifetime and multiple borrow compiler errors with the mutable version.

I would like to know if the work-around I chose is sound and idiomatic (as much as it can be for a work-around), and if there isn't a side-effect I overlooked.

I've put a simplified version here, with only one edge to the next item: playground. Please bear in mind that the actual tree structure has multiple edges and other features, so the idea isn't to refactor the code shown here to use slice iterators, for example.

The naive, first attempt looked like this:

// The list structure:
pub struct RevList<T> {
    data: Vec<T>,               // items
    next: Vec<Option<usize>>,   // next[i] is the index of data[i]'s next item
    head: Option<usize>         // index of first item in the list
}

// The mutable iterator:
pub struct RLIterMut<'a, T> {
    next: Option<usize>,
    list: &'a mut RevList<T>
}

impl<'a, T> Iterator for RLIterMut<'a, T> {
    type Item = &'a mut T;

    fn next(&mut self) -> Option<Self::Item> {
        if let Some(index) = self.next {
            self.next = self.list.next[index];
            Some(&mut self.list.data[index])    // <=== ERROR
        } else {
            None
        }
    }
}

which generates error: lifetime may not live long enough since the lifetime of &mut self doesn't equal the expected &'a mut T of the return type. There is also the hidden problem that the compiler can't tell if one or several &mut references to the same items are returned (this error can be seen by using a custom iterator trait to work around the 'a lifetime issue).

For the safety: in my code, both here and in the real project, I'm sure that each item reference is returned only once by the iterator. I can also tell the lifetime of self.list.data is the same as self because I don't drop anything.

Here are the two alternatives I have, to replace the faulty line shown above:

            let value: &mut T = &mut self.list.data[index];

            // 2 alternatives:
            let ptr: &mut T = unsafe { 
                std::mem::transmute::<&mut T, &'a mut T>(value) 
            };
            let ptr: &mut T = unsafe { 
                &mut *(value as *mut T) 
            };

            Some(ptr)

The transmute seems the better choice to me since it allows to explicit the lifetime conversion.

It would also be possible to store raw pointers in the iterator, but the idea is the same as the reference cast alternative above: making the lifetime disappear with raw pointers in unsafe code.

Which is better? Is there another, better way to deal with this?

Thanks. :slight_smile:

I haven’t read beyond this point yet and haven’t looked into the code at all, but simply by rewriting the code in main

    for item in l.iter_mut() {
        *item *= 2;
    }

to keep iterator items around in parallel

    let items = l.iter_mut().collect::<Vec<_>>();
    for item in items {
        *item *= 2;
    }

miri reports undefined behavior. So it’s not sound.

error: Undefined Behavior: trying to retag from <6933> for Unique permission at alloc1088[0x8], but that tag does not exist in the borrow stack for this location
   --> /playground/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/vec/into_iter.rs:202:27
    |
202 |             Some(unsafe { ptr::read(old) })
    |                           ^^^^^^^^^^^^^^
    |                           |
    |                           trying to retag from <6933> for Unique permission at alloc1088[0x8], but that tag does not exist in the borrow stack for this location
    |                           this error occurs as part of retag at alloc1088[0x8..0xc]
    |
    = help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental
    = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
help: <6933> was created by a Unique retag at offsets [0x8..0xc]
   --> src/main.rs:95:17
    |
95  |     let items = l.iter_mut().collect::<Vec<_>>();
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
help: <6933> was later invalidated at offsets [0x0..0xc] by a Unique retag
   --> src/main.rs:72:52
    |
72  |             let value: &mut T = &mut self.list.data[index];
    |                                                    ^^^^^^^
    = note: BACKTRACE (of the first span):
    = note: inside `<std::vec::IntoIter<&mut i32> as std::iter::Iterator>::next` at /playground/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/vec/into_iter.rs:202:27: 202:41
note: inside `main`
   --> src/main.rs:96:17
    |
96  |     for item in items {
    |                 ^^^^^
3 Likes

Using unsafe and/or transmute to work around lifetime errors almost invariably means that you have written unsound code and it has undefined behavior. It's certainly not idiomatic, and most probably it's just outright wrong.

If you want a mutable iterator over slices, then your best bet is to wrap core::slice::IterMut.

No, you should basically never transmute pointers. Their layout is not guaranteed (especially not for fat pointers). You should be performing the explicit address-of and/or as cast instead.

3 Likes

The issue can be solved (still using unsafe code, but miri is happier) using a raw pointer instead of &mut reference. I.e. to obtain multiple mutable references to Vec elements, not working with a &mut Vec<T> or &mut StructHoldingVec<T>, but instead a raw pointer *mut T pointing to the Vec’s data, and using pointer arithmetic and dereferencing to access the element in a way that doesn’t go against Rust’s model of reference aliasing / uniqueness around &mut …

An alternative to using *mut T for the slice can also be to use &[Cell<T>] (with its unsafe UnsafeCell-style API, whilst Cell has the advantage of having more convenient constructors from_mut and as_slice_of_cells), which allows using indexing operations again, and skips the need for a marker: PhantomData<…> part.

Either way, in each case, of course (though I haven’t commented in the playgrounds I’ve linked) the safety-relevant thing for those unsafe blocks still is that all accessed indices are unique by construction.

That's interesting. I tested a few out-of-scope uses but didn't meet any problem. I didn't try this, though.

The same code can be used on simple vectors, yet it doesn't trigger an error with Miri:

fn test3() {
    let mut l = vec![1, 2, 3];
    let items = l.iter_mut().collect::<Vec<_>>();
    for item in items {
        *item *= 2;
        println!("{}", *item);
    }
}

The code in the standard library is hidden in several layers of macros, but they're using raw pointers in safe code, so I don't see the difference.

On the other hand, the report from Miri mentions that it's a potential error out of an experimental rule, so it may be a false positive.

If we look at what happens, the collect may be seen as an immutable reference access to the &mut provided by the iterator, which could trigger that sort of error. But in reality, is there really a problem here? Each item only has one mutable reference.

If now, I change the code to:

    let mut l = vec![1, 2, 3];
    let items = l.iter_mut().collect::<Vec<_>>();
    let r1 = &items[1];
    *items[1] *= 2;
    println!("{}", *r1);

It won't compile because the compiler still detects the problem:

error[E0502]: cannot borrow `items` as mutable because it is also borrowed as immutable
   --> src\list3_mut.rs:115:10
    |
114 |         let r1 = &items[1];
    |                   ----- immutable borrow occurs here
115 |         *items[1] *= 2;
    |          ^^^^^ mutable borrow occurs here
116 |         println!("{}", *r1);
    |                        --- immutable borrow later used here

The mutable reference has simply moved from the iterator to the vector.

So this doesn't tell me it's unsound. I suppose Miri doesn't see the problem with a native vector because the code is already native and not interpreted.

They are references of the same type, except the lifetime which is a compiler artefact that disappears after compilation. Is there really a risk to see a difference here?

For example, if you look at the code with Compiler Explorer, the produced code is exactly the same with the 2 alternatives. Nothing actually happens and this cast / transmute is removed from the assembly code.

But I understand that transmute does simply interpret the content as is, so there is a risk when data are not aligned, and so on.

Vector iterators (or more precisely slice iterators) use raw pointers to the slice data, not a &'a mut Vec<T> field. Their approach is comparable to the approach with a list_data: *mut T field.

It’s not a false positive. Miri should not have any false positives. It’s correctly reporting that it’s hitting a case of “undefined behavior” where it’s not yet decided whether this will be considered actual “undefined behavior” in the Rust language design, or eventually be allowed. Call it “behavior that’s not yet defined whether it’s considered undefined”.

Besides the question of whether or not obtaining co-existing &mut T references to Vec elements via the ordinary IndexMut interface triggers true language-level undefined behavior, there’s also nothing in the standard library docs that promises the behavior of the IndexMut implementation will always stay this way that it “only” triggers “maybe no longer undefined behavior in the future” corner-cases.

Either way, I see no convincing reason whatsoever to write the code like this, relying on edge-case language specificities and undocumented standard library behavior, when (as I’ve demonstrated, even with two alternative approaches) better alternatives with none of those shortcomings clearly exist.

1 Like

That's also what I saw. And as you say, the flow is slightly different (as seen by Miri) because we increment a raw pointer instead of casting a reference by using a raw pointer.

But in practice, is there really a difference, and is the pointer approach safer? I'm asking because it will make the code much more unreadable in my real application.

I still have to check your two solutions in more details, thanks a lot for all that! :slight_smile:

Generated machine code does not constitute proof of correctness regarding the high-level language specification.

1 Like

Feel free to come back with follow-up questions in case the approaches both don’t translate to your non-simplified use-case.

1 Like

It's not, I mentioned that as an example of what transmute does. I also said that transmute (re)interpreted the content as is, so without modification. I just don't see the risk. It's actually one of the use-cases in the documentation.

But thanks, I take notice that transmute looks like a potential risk to developers, so perhaps it's best to avoid it just for that reason, even if in this case I fail to see the problem. Or it should at least have a comment with the proper justification.

Unfortunately, as I said explicitly in the first post, this is only a simple example and I can't use iterators over slices in the real code. :slight_smile:

EDIT:

I missed that earlier, and I find it surprising.

The iter_mut is a well-known case that cannot be solved otherwise than using unsafe code (EDIT: in many cases, but not all), directly or indirectly. The compiler is simply unable to verify the lifetime and the borrow soundness, so this has to be performed by the developer. If you look in the standard library, you'll see that unsafe code is used for those iterators, and yet it's not outright wrong. :wink:

Not really true. Demo.

The standard library was written by the very inventors of the language and has contributions from those working on the compiler, the language design, and the operational semantics, and it has been subject to constant scrutiny during the last 16 years. Also, std is free to make assumptions about the internals of the compiler and the language (esp. w.r.t. yet-unspecified aspects of opsem) that no 3rd-party code gets to make.

I'm sorry but there is a real effect here. When people think they need to use unsafe, in the overwhelming majority of cases they really don't, because either what they want to do is disastrously wrong, or it has been done by the standard library in a much better and safer way. I've been active on this forum for ~7 years, and it is positively the case that the vast majority of "but trust me I need unsafe" assertions ends up doing something fishy at best, or clearly unsound at worst. If you need to ask whether some unsafe you wrote is sound, then you are probably not prepared to write it in the first place.

3 Likes

FYI no, the stdlib is also interpreted by MIRI and some bugs have been found in it thanks to that.

3 Likes

Thanks, but none of that is helping.

Yes, there are cases that don't need unsafe. Do you mean that you see a solution for my case that can be solved without it? Or do you see a real problem in that code, other than something potential because there is unsafe and/or transmute? That's what I'm trying to work out.

And surely you don't mean that unsafe cannot be used. There are many situations where it's useful, like this one.

Good to know!

I'll have to understand where the problem is triggered (the first message is related to a line number outside the scope of my program) and why it is a problem (if it is a real one). Or ignore it entirely and use the raw pointer approach.

The soundness depends on the logic in append. Phrased better, the soundness of the iterator depends on the absence of logic errors inside append, which would be very hard to prove.

2 Likes

Do you mean regarding the fact each index value is unique?

As an alternative, if I remove the let value ... and only replace the transmute by this code, which is based on your first suggestion, it doesn't trigger any issue in Miri either:

impl<'a, T> Iterator for RLIterMut<'a, T> {
    type Item = &'a mut T;

    fn next(&mut self) -> Option<Self::Item> {        
        if let Some(index) = self.next {
            self.next = self.list.next[index];
            // let value: &mut T = &mut self.list.data[index];
            // let ptr = unsafe { std::mem::transmute::<&mut T, &'a mut T>(value_ptr) };
            // let ptr = unsafe { &mut *(value as *mut T) };
            let ptr: &mut T = unsafe {
                let value_ptr = self.list.data.as_mut_ptr().add(index);
                &mut *value_ptr
            };

            Some(ptr)
        } else {
            None
        }
    }
}

I still need to understand if there's a real difference between the two. Either I'm missing something, or it may be an interesting case for Miri.

In any case, your pointer suggestion is the best way to proceed and it's simple enough, so I marked it as solution. The PhantomData is a very helpful reminder too, because without it, the error message of the compiler may not be straightforward to solve without its knowledge.

Thanks again!

Yes. It would be unsound if there would be duplicate entries in next. In that case, the iterator's next would hand out multiple mutable references to the same value. I think your code could be implemented with RefCell, which would panic in such a situation. I'd prefer that over unsafe.

1 Like