Soundness of shortening the lifetime of a mut ref

[zslayton]

Here's a complete playground example of the code.

I have a struct that encodes values into an owned buffer. It uses the bumpalo crate's Bump and Vec<'a, T> types, locally aliasing them Bump -> BumpAllocator and Vec -> BumpVec.

struct ValueWriter<'value> {
    // We have this reference to enable the creation of more buffers,
    // out of scope for this problem but part of the data structure.
    allocator: &'value BumpAllocator,
    // Our Vec only contains unsigned bytes
    buffer: &'value mut BumpVec<'value, u8>
}

I would like write an accessor method that lends out a &mut reference to the buffer field, but which lends it out with a lifetime that is shorter than 'value.

If I write a method like this:

fn buffer(&'value mut self) -> &'value BumpVec<'value> {
  self.buffer
}

then any caller would tie up the ValueWriter for the entirety of 'value; I can't do anything else to self.

let mut writer = ValueWriter::new(...);
let l = writer.buffer().len(); // contrived example
writer.do_stuff(); // ERROR; `writer` cannot be borrowed a second time

If I could instead write a method with this signature:

pub fn buffer(&mut self) -> &'_ mut BumpVec<'_, u8> {
  todo!()
}

...I'd be able to access the buffer as needed in the parent scope without locking up self.

Having stubbed my toe on a series of errors discussing invariance, I took the time to try and better understand the reference's chapter on Subtyping and Variance, which gives this example:

I think that it's saying that &mut T cannot do lifetime subtyping because it would be possible to assign a short-lived value to a field in the longer-lived struct, which would lead to undefined behavior when you went back to working with the longer-lived struct.

What I'm unclear on is whether my unsafe code is hazardous given that my struct's only mutable field is a BumpVec<'value, u8>, which only stores values that are Copy/'static (u8). It seems like in this case it might be alright; I'm able to implement the method above with unsafe casting and it's working (for now!) but I'd like the opinion of someone with a stronger grasp of variance and UB.

Thanks for any help!

EDIT: I should clarify: I know that it's possible to achieve the above safely if I modify ValueWriter to use multiple lifetimes. However, the type is very central to my API and is used in GATs which currently introduces several sharp edges I'm looking to sidestep. If my unsafe approach is dangerous, I'd like to understand why.

[kpreid]

@zslayton

whether my unsafe code is hazardous given that my struct's only mutable field is a BumpVec<'value, u8>, which only stores values that are Copy/'static (u8).

The lifetime on the BumpVec is not just about the data, it's the lifetime of the BumpVec's use of the allocator. So, someone could use this returned &mut reference to swap out the BumpVec for one which uses a shorter-lived allocator, thus causing ValueWriter to use-after-free the next time it's used.

Always remember that giving out &mut T is permission to replace the T with a totally different value of the same type.

Instead of unsound lifetime manipulation, what you should do is stop conflating the two roles of lifetimes in your struct. &'value mut BumpVec<'value, u8> — in general, &'a mut T<'a> — is essentially always incorrect because it contains lifetime constraints that end up meaning that the BumpVec is borrowed for the rest of its existence as soon as you create one such &mut reference, and it cannot even be reborrowed for less time than that ('value becomes an invariant lifetime). The fix is to use two lifetime parameters:

struct ValueWriter<'vec, 'bump> {
    allocator: &'bump BumpAllocator,
    buffer: &'vec mut BumpVec<'bump, u8>
}

impl<'vec, 'bump> ValueWriter<'vec, 'bump> {
    pub fn buffer(&mut self) -> &'_ mut BumpVec<'bump, u8> {
        &mut self.buffer
    }
}

This does what you seem to be asking for:

I would like write an accessor method that lends out a &mut reference to the buffer field, but which lends it out with a lifetime that is shorter than 'value .

It lends it out for the lifetime of the &mut self, which is whatever the caller wants it to be (as long as it's shorter than both 'vec and 'bump).

[zslayton]

So, someone could use this returned &mut reference to swap out the BumpVec for one which uses a shorter-lived allocator, thus causing ValueWriter to use-after-free the next time it's used.
Always remember that giving out &mut T is permission to replace the T with a totally different value of the same type.

This was the gap in my mental model, thank you!

This does what you seem to be asking for:
It lends it out for the lifetime of the &mut self, which is whatever the caller wants it to be (as long as it's shorter than both 'vec and 'bump).

You're correct, it does. Unfortunately, the problem I'm running into is downstream of this. There's a trait Foo that includes something like:

trait Foo {
    // ...
    type ListWriter<'a>;
    // ...
    fn write_list<F>(
        self,
        list_fn: F,
    ) -> MyResult<()>
      where
        F : for<'a> FnOnce(&mut Self::ListWriter<'a>) -> MyResult<()>;
}

Due to a limitation in the borrow checker and the current implementation of GATs, there are expressiveness restrictions that I'm not able to work around when there are multiple lifetimes in the HRTB. I'm looking for a short-term way to skirt this problem. I started by seeing if I could just shorten 'value to '_ via unsafe shenanigans, but I'd love to find something less hacky/dangerous.

[kpreid]

So this doesn't work?

impl<'bump> Foo for MyFooImplementor<'bump> {
    type ListWriter<'a> = ValueWriter<'a, 'bump>;
    ...

[zslayton]

I believe the compiler complained about being unable to guarantee that 'a outlived 'bump and/or Self. Trying to (partially?) address the problem by adding where Self: 'a led to other point-in-time limitations in GATs. I was not able to find an HRTB syntax that would support expressing a relationship between two lifetime parameters.

I'm no longer near a computer and will have to get you a more satisfying explanation tomorrow. Thank you for your help thus far!

[quinedot]

You may be able to:

trait StuffYouNeedValueWriterToDo {
    fn stuff(&mut self) {}
}

impl StuffYouNeedValueWriterToDo for ValueWriter<'_, '_> {}

impl<'bump> Foo for MyFooImplementor<'bump, u8> {
    // If `ListWriter<'a>: ?Sized
    type ListWriter<'a> = dyn StuffYouNeedValueWriterToDo + 'a;
    // Else
    // type ListWriter<'a> = &'a mut dyn StuffYouNeedValueWriterToDo;
}

Or there might be something better, but it probably depends on exactly what the trait and implementor look like.

[zslayton]

Following up on this, I managed to get my code working by counter-intuitively ignoring the <'a> lifetime on the associated type.

impl<'value, 'bump> ValueWriter for MyValueWriter <'value, 'bump> {
    //               ┌ this lifetime...
    type ListWriter<'a> = MyListWriter<'value, 'bump> ;
    // ...is not used in the type. ───┴─────────────┘

    // ...methods...
}

(Playground version)

The associated type's lifetime 'a was added because I didn't want the closure to be mutably borrowed (and therefore locked up) for the entirety of lifetime 'value.

    pub fn write_list<
        F: FnOnce(&mut <Self as ValueWriter>::ListWriter) -> IonResult<()>,
    >(
        self,
        list_fn: F,
    ) -> IonResult<()> {
        // Once `list_fn` is called here, `self` borrowed for the rest of
        // its existence.
    }

I was hoping to use HRTB to only borrow it for the duration of the closure, but HRTB lifetimes cannot be bound in relation to other lifetimes. Because I couldn't convey that 'value: 'a, the compiler would reject the method+closure signature.

I can workaround the lockup problem for now, so I'm unstuck. Thanks everyone for your help/input!