Cursed, Pinned Cells

We like to believe UnsafeCell<T> is opaque to its contents and returns an arbitrary *mut T to its contents. This is probably not aligned with the existing Stacked Borrows model, but it's just a model and those can be changed. Additionally, for convenience, UnsafeCell<T> contains a helper fn get_mut(&mut self) -> &mut T which ties the output reference's lifetime to the input reference's lifetime, while preventing simultaneous access to the UnsafeCell<T> as that would be unsound. (Our Cursed Cells won't have that helper, as it would be unsound.) Indeed, this all aligns well with how Miri doesn't complain about it: Rust Playground

So we have our CursedCell:

#[repr(transparent)]
pub struct CursedCell<T: ?Sized> {
  inner: Cell<T>,
}

impl<T> CursedCell<T> {
  pub fn new(t: T) -> Self {
    Self { inner: Cell::new(t) }
  }
}

impl<T: ?Sized> CursedCell<T> {
  pub fn as_ptr(&self) -> *mut T {
    self.inner.as_ptr()
  }
}

Now, the space held by the cell is not actually forever. We'd love to represent that in the type system, but we don't know how to. Tho we can do this: Rust Playground And there is one place where Rust infers lifetimes but you can otherwise override it: dyn Trait. Specifically:

type Foo<'a> = &'a dyn Trait;
// same as:
type Foo<'a> = &'a (dyn Trait + 'a);

// but you can also do:
type Foo<'a> = &'a (dyn Trait + 'static);

// while at it:
type Foo = Box<dyn Trait>; // implies 'static

Unfortunately we can't make use of this. But we might not need it anyway.

The struct above is immovable after throwing &foo into it, as it is borrowed. (Tho it cannot be moved again even after setting x to None!) But it can be moved before then. Let's get rid of the Cell first: Rust Playground

struct Foo<'a> {
    x: Option<&'a Foo<'a>>,
}

fn main() {
    let mut foo = Foo {
        x: None,
    };
    let x = &mut foo.x;
    *x = Some(&foo);
}

Alright, this clearly doesn't work. But all is not lost. Now, we can't safely do cell projections, but they're still sound. So let's use a Cell, and change that x to have a &Cell<Foo> instead:

use std::cell::Cell;
struct Foo<'a> {
    x: Option<&'a Cell<Foo<'a>>>,
}

fn main() {
    let foo = Cell::new(Foo {
        x: None,
    });
    let x: &Cell<Option<&Cell<Foo<'_>>>> = todo!();
    x.set(Some(&foo));
}

We can also Box::pin it. Tho on that note std makes it really hard to work with pinned cells, so let's use our CursedCell instead, and add things to make working with Pin easier. Maybe even fork pin-project so it works with the CursedCell. Now, it's important to note that with pinned types, standard Cell is unsound (e.g. the example above is unsound), because you could set the whole Foo, while Cell moves it: Rust Playground While at it, standard Drop is also unsound: Rust Playground (run it in miri) Ideally we'd deal with that in CursedCell somehow, maybe by emitting our own drop glue (and having a CursedDrop which gets Pin<&'a CursedCell<Self<'a>>>) in the #[cursed_pin] macro, and using a ManuallyDrop (which is, like Cell, #[repr(transparent)], so shouldn't affect projections) inside the CursedCell. This would also have to be guarded against by any safe, non-pinned "cell projection" crates.

Anyway sorry, we burned out at this point in writing, so we're not gonna finish this post. We just wanted an excuse to talk about stacked borrows and Pin and Cell. Stacked borrows as currently implemented in miri does seem to make Cell's contents effectively live for as long as the Cell isn't moved, regardless of how the borrows leading up to the *mut T are tracked, so that's nice. It'd be nice if there were a crate that made use of it.

Note that when dealing with raw pointers, you usually want to use -Zmiri-tag-raw-pointers (or Zmiri-strict-aliasing if you don't need usize to pointer casts). Unfortunately the playground doesn't allow setting Miri hardening flags , but they would make your initial example diagnose UB.

2 Likes

we think everyone can agree that the initial example isn't UB, and shouldn't cause miri to complain.

the hardening flags are unstable for a reason. (well miri is unstable for a reason. what does LLVM think of this whole thing?)

(... also why doesn't rustc complain about the implicit borrows caused by drop_in_place it introduces as part of drop glue? edit: oh it complains if you have an actual Drop, interesting! - tho it makes us wonder why cell projection, particularly with the cell-project crate, doesn't make this error show up.)

also, if we did it correctly, this doesn't complain:

use std::cell::UnsafeCell;

struct CursedCell<T> {
    inner: UnsafeCell<T>,
}

impl<T> CursedCell<T> {
    fn new(t: T) -> Self {
        Self {
            inner: UnsafeCell::new(t),
        }
    }
    fn get_mut(&mut self) -> &mut T {
        unsafe { &mut *self.get() }
    }
    fn get(&self) -> *mut T {
        self.inner.get()
    }
}

fn main() {
    let mut foo = CursedCell::new(0);
    let ptr = foo.get();
    unsafe { *ptr = 1; }
    *foo.get_mut() = 2;
    unsafe { *ptr = 3; }
}

Apologies, apparently I misremembered how exactly &mut UnsafeCell behaves. I still think it causing an error would be a valid interpretation of the rules, but miri disagrees with me here.

Anyway, the best way to handle stack pinning is the pin! macro (or on stable, pin_mut!). Self-pinning with self-borrowing lifetimes is cute, but ultimately not really more useful than the existing stack pinning practice.

It doesn’t take much change for miri to start complaining.

use std::cell::UnsafeCell;

fn main() {
    let mut cell = UnsafeCell::new(0);
    let ptr = cell.get();
    unsafe { *ptr += 1; }

    // 3 alternatives:

    *cell.get_mut() += 1; // miri won't complain
    // *(&mut cell).get_mut() += 1; // miri complains
    // *UnsafeCell::get_mut(&mut cell) += 1; // miri complains

    unsafe { *ptr += 1; }

    dbg!(cell.get_mut());
}

I couldn’t tell you what the relevant difference here should be to justify the change in behavior [1], maybe a bug and miri should really complain in all 3 cases, after all?


  1. possibly the latter 2 cases introduce an additional step of reborrowing the mutable reference created by auto-ref? ↩︎

4 Likes

I opened an issue at the UCG repo to clarify this behavior.

1 Like

Does this still happen with the CursedCell get_mut?

Yes, but with CursedCell it only complains (with the explicit &mut) when -Zmiri-tag-raw-pointers is turned on, whereas the original example (with the additional explicit &mut) complains with default miri settings, too.

Hmm...

Eh at least it doesn't seem to be important for all the other examples. Tho we still maintain that UnsafeCell should be opaque to its contents.


It'd be nice to have this error:

use std::cell::Cell;
use cell_project::cell_project;

struct Foo<'a> {
    x: Option<&'a Cell<Foo<'a>>>,
}

fn main() {
    let foo = Box::new(Cell::new(Foo {
        x: None,
    }));
    let x = cell_project!(Foo<'_>, foo.x);
    x.set(Some(&*foo));
    let bar = foo;
}

but have this compile:

use std::cell::Cell;
use cell_project::cell_project;

struct Foo<'a> {
    x: Option<&'a Cell<Foo<'a>>>,
}

fn main() {
    let foo = Box::pin(Cell::new(Foo {
        x: None,
    }));
    let x = cell_project!(Foo<'_>, foo.x);
    x.set(Some(&*foo));
    let bar = foo;
}

but it'd probably require HKTs or something.

see, we just need this to be part of std: Rust Playground

then again, miri doesn't seem to complain. (we're not sure how to get miri to complain?)


or we guess we can do something like this? Rust Playground


also miri doesn't complain about this, despite the above UCG issue by @CAD97 which would seem related: (sadly we seem to be blocked from that particular issue so we can't bring it up there)

use std::cell::Cell;
use std::marker::PhantomPinned;
use std::pin::Pin;

use cell_project::cell_project;

struct Foo<'a> {
  x: Option<&'a Cell<Foo<'a>>>,
  y: usize,
}

impl<'a> Foo<'a> {
    // can a custom trait mirror the observable behaviour of Drop? (on nightly?)
    fn fake_drop(this: Pin<&'a Cell<Foo<'a>>>) {
        let bar = cell_project!(Foo<'_>, this.x);
        let inner = bar.get().unwrap();
        println!("{:p}", this.get_ref());
        println!("{:p}", inner);
        println!("{}", cell_project!(Foo<'_>, inner.y).get());
        cell_project!(Foo<'_>, inner.y).set(3);
    }
}

// not allowed to implement this
// impl<'a> Drop for Foo<'a> {
//     fn drop(&mut self) {
//     }
// }

// want: macro that does all this:

#[repr(transparent)]
struct FooOpaque {
  // unsafe field: not actually 'static
  inner: Foo<'static>,
  _pinned: PhantomPinned,
}

impl FooOpaque {
    fn new(foo: impl for<'a> FnOnce(&'a ()) -> Foo<'a>) -> Self {
        #[allow(unreachable_code)]
        fn _prevent_problematic_drop(f: impl for<'x> Fn(&'x ()) -> Foo<'x>) {
            let _arg = ();
            let _foo: Foo<'_> = f(&_arg);
            let _cell = Cell::new(_foo);
            #[inline(never)]
            fn _f<'a>(_x: &'a Cell<Foo<'a>>) {}
            _f(&_cell);
        }
        Self {
            inner: unsafe { std::mem::transmute(foo(&())) },
            _pinned: PhantomPinned,
        }
    }
    
    // this MUST NOT be a Cell because those have set(), replace(), swap(),
    // etc. but this here is just an example.
    fn operate_in<F, R>(pinned_cell: Pin<&Cell<Self>>, f: F) -> R
    where F: for<'a> FnOnce(Pin<&'a Cell<Foo<'a>>>) -> R {
        f(unsafe {
            pinned_cell.map_unchecked(|cell_ref| {
                std::mem::transmute(cell_project!(FooOpaque, cell_ref.inner))
            })
        })
    }
}

impl Drop for FooOpaque {
    fn drop(&mut self) {
        unsafe {
            // assume it was pinned.
            Self::operate_in(std::mem::transmute(self), |cell_ref| {
                Foo::fake_drop(cell_ref)
            });
        }
    }
}

fn now_can_move() {
    let foo = Box::pin(Cell::new(FooOpaque::new(|_| Foo {
        x: None,
        y: 0,
    })));
    FooOpaque::operate_in(foo.as_ref(), |foo| {
        let bar = cell_project!(Foo<'_>, foo.x);
        bar.set(Some(foo.get_ref()));
        println!("{}", cell_project!(Foo<'_>, foo.y).get());
        cell_project!(Foo<'_>, foo.y).set(1);
    });
    FooOpaque::operate_in(foo.as_ref(), |foo| {
        let bar = cell_project!(Foo<'_>, foo.x);
        let inner = bar.get().unwrap();
        println!("{:p}", foo.get_ref());
        println!("{:p}", inner);
        println!("{}", cell_project!(Foo<'_>, inner.y).get());
        cell_project!(Foo<'_>, inner.y).set(2);
    });
}

fn main() {
    now_can_move();
}

so it seems we actually have safe* pinned with self-references. any objections?

(* need to get rid of some of those Cell and replace them with CursedCell.)


so yeah, how do we make a proc macro that takes this

struct Foo<'self> {
    x: Option<&'self SelfCell<Foo<'self>>>,
}

impl<'self> Drop for Foo<'self> {
    fn drop_pin(this: Pin<&'self SelfCell<Foo<'self>>>) {
        ...
    }
}

and generates all this

struct Foo<'a> {
    x: Option<&'a SelfCell<Foo<'a>>>,
}

#[repr(transparent)]
struct FooOpaque {
    // unsafe field: not actually 'static
    inner: Foo<'static>,
    _pinned: PhantomPinned,
}

impl FooOpaque {
    fn new(foo: impl for<'a> FnOnce(&'a ()) -> Foo<'a>) -> Self {
        #[allow(unreachable_code)]
        fn _prevent_problematic_drop(f: impl for<'x> Fn(&'x ()) -> Foo<'x>) {
            let _arg = ();
            let _foo: Foo<'_> = f(&_arg);
            let _cell = Cell::new(_foo);
            #[inline(never)]
            fn _f<'a>(_x: &'a Cell<Foo<'a>>) {}
            _f(&_cell);
        }
        Self {
            inner: unsafe { std::mem::transmute(foo(&())) },
            _pinned: PhantomPinned,
        }
    }
    
    // this MUST NOT be a Cell because those have set(), replace(), swap(),
    // etc.
    fn operate_in<F, R>(pinned_cell: Pin<&SelfCell<Self>>, f: F) -> R
    where F: for<'a> FnOnce(Pin<&'a SelfCell<Foo<'a>>>) -> R {
        f(unsafe {
            pinned_cell.map_unchecked(|cell_ref| {
                std::mem::transmute(cell_project!(FooOpaque, cell_ref.inner))
            })
        })
    }
}

impl Drop for FooOpaque {
    fn drop(&mut self) {
        fn drop_pin<'a>(this: Pin<&'a SelfCell<Foo<'a>>>) {
            ...
        }
        unsafe {
            // assume it was pinned.
            Self::operate_in(std::mem::transmute(self), |cell_ref| {
                drop_pin(cell_ref)
            });
        }
    }
}

(anyone knows why miri doesn't complain about that one with the FooOpaque and the Drop and the Cell btw?)

I believe what you're running into is

!Unpin currently has a hack applied to it to treat &mut impl !Unpin as &impl !Unpin to avoid causing issues due to the above. This removes the noalias in LLVM and the uniqueness checks in miri.

This is even more not guaranteed than the rest of Stacked Borrows, and is very explicitly a stopgap to avoid miscompiling async. This has a high possibility of changing in the future[1], and definitely should not be relied upon.

Remember: miri does not prove the absence of UB (yet).


  1. A recently discussed refinement is to treat it more granularity like UnsafeCell. Currently, the "!Unpin hack" applies to any type which is !Unpin (transitively contains PhantomPinned). The proposal is to refine the application of the looser semantics to just the region marked with "UnsafeMutCell", just like UnsafeCell only allows writing to the region contained in an UnsafeCell and not any type which is !Freeze (internal implementation detail for transitively contains UnsafeCell). This playground explains by example. ↩︎

1 Like

the presence or absence of _pinned doesn't seem to affect whether or not miri complains about it. as such, can we ship it yet?

use std::cell::Cell;
use std::marker::PhantomPinned;
use std::pin::Pin;

use cell_project::cell_project;

struct Foo<'a> {
  x: Option<&'a Cell<Foo<'a>>>,
  y: usize,
}

impl<'a> Foo<'a> {
    // can a custom trait mirror the observable behaviour of Drop? (on nightly?)
    fn fake_drop(this: Pin<&'a Cell<Foo<'a>>>) {
        let bar = cell_project!(Foo<'_>, this.x);
        let inner = bar.get().unwrap();
        println!("{:p}", this.get_ref());
        println!("{:p}", inner);
        println!("{}", cell_project!(Foo<'_>, inner.y).get());
        cell_project!(Foo<'_>, inner.y).set(3);
    }
}

// not allowed to implement this
// impl<'a> Drop for Foo<'a> {
//     fn drop(&mut self) {
//     }
// }

// want: macro that does all this:

#[repr(transparent)]
struct FooOpaque {
  // unsafe field: not actually 'static
  inner: Foo<'static>,
  //_pinned: PhantomPinned,
}

impl FooOpaque {
    fn new(foo: impl for<'a> FnOnce(&'a ()) -> Foo<'a>) -> Self {
        #[allow(unreachable_code)]
        fn _prevent_problematic_drop(f: impl for<'x> Fn(&'x ()) -> Foo<'x>) {
            let _arg = ();
            let _foo: Foo<'_> = f(&_arg);
            let _cell = Cell::new(_foo);
            #[inline(never)]
            fn _f<'a>(_x: &'a Cell<Foo<'a>>) {}
            _f(&_cell);
        }
        Self {
            inner: unsafe { std::mem::transmute(foo(&())) },
            //_pinned: PhantomPinned,
        }
    }
    
    // this MUST NOT be a Cell because those have set(), replace(), swap(),
    // etc. but this here is just an example.
    fn operate_in<F, R>(pinned_cell: Pin<&Cell<Self>>, f: F) -> R
    where F: for<'a> FnOnce(Pin<&'a Cell<Foo<'a>>>) -> R {
        f(unsafe {
            pinned_cell.map_unchecked(|cell_ref| {
                std::mem::transmute(cell_project!(FooOpaque, cell_ref.inner))
            })
        })
    }
}

impl Drop for FooOpaque {
    fn drop(&mut self) {
        unsafe {
            // assume it was pinned.
            Self::operate_in(std::mem::transmute(self), |cell_ref| {
                Foo::fake_drop(cell_ref)
            });
        }
    }
}

fn now_can_move() {
    let foo = Box::pin(Cell::new(FooOpaque::new(|_| Foo {
        x: None,
        y: 0,
    })));
    FooOpaque::operate_in(foo.as_ref(), |foo| {
        let bar = cell_project!(Foo<'_>, foo.x);
        bar.set(Some(foo.get_ref()));
        println!("{}", cell_project!(Foo<'_>, foo.y).get());
        cell_project!(Foo<'_>, foo.y).set(1);
    });
    FooOpaque::operate_in(foo.as_ref(), |foo| {
        let bar = cell_project!(Foo<'_>, foo.x);
        let inner = bar.get().unwrap();
        println!("{:p}", foo.get_ref());
        println!("{:p}", inner);
        println!("{}", cell_project!(Foo<'_>, inner.y).get());
        cell_project!(Foo<'_>, inner.y).set(2);
    });
}

fn main() {
    now_can_move();
}

I haven't read this entire thread, but your Foo struct is Unpin so pinning it is a no-op and does nothing.

1 Like

see FooOpaque - which holds a self-referential struct with lifetimes, but attempts to hide said lifetime from the type system.

but we guess you do have a point in that we don't need the Pin on the closure to operate_in...

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.