Creating struct in shared memory

help
1

Hi,

I'm trying to port the shared memory example in shm_open man page to Rust. I've defined the shmbuf type like so:

pub struct Shmbuf<const N: usize> {
    sem1: Semaphore,
    sem2: Semaphore,
    buf: [u8; N],
}

impl<const N: usize> Shmbuf<N> {
   pub fn new(mem: &mut Mmap) -> Result<&mut Self> {
        assert!(
            mem.len() >= size_of::<Self>(),
            "size of shared memory segment cannot be smaller than Shmbuf"
        );
        let shmbuf = mem.as_mut_ptr() as *mut Shmbuf<N>;
        unsafe {
            (&raw mut (*shmbuf).sem1).write(Semaphore::new(0)?);
            (&raw mut (*shmbuf).sem2).write(Semaphore::new(0)?);
            (&raw mut (*shmbuf).buf).write([0; N]);
            Ok(&mut *shmbuf)
        }
    }

    pub fn from_shm_mut(mem: &mut Mmap) -> Result<&mut Self> {
        let shmbuf = mem.as_mut_ptr() as *mut Shmbuf<N>;
        unsafe { Ok(&mut *shmbuf) }
    }
}

where Semaphore is a thin wrapper around sem_t. The bounce program creates the shared memory and calls new, and the send program opens the shared memory and calls from_shm_mut.

This works, but I'm not sure if I'm violating the rules of unsafe Rust, specifically:

  • It didn't add repr(c) to the struct definition. Am I making faulty assumptions about the layout of Shmbuf?
  • I get a mutable pointer to shared memory, and then get three more mutable pointers into the same memory, which seems like it violates Rust's aliasing rules.
  • I don't know if memory allocated with mmap is considered initialized.
  • The Shmbuf struct is never dropped, which makes sense so far as I think it's tied to the lifetime of mem, but a problem if I mistakenly thought the Semaphores were being dropped.

Am I on the right track with the rules as I understand them? And how can I do the same thing in a safer way?

Thanks,
Andy

2

Rust compiler reserves right to make layout of structs random, so you absolutely need repr(C) on anything that needs to work with other executables.

You're not allowed to make a reference to memory that can be changed from outside of the process. Rust assumes it will never happen. &mut can only be mutated by Rust.

I don't know if Mmap wraps the data in UnsafeCell for you. The shared memory should be in an UnsafeCell, which opts out of Rust's aliasing guarantees, and you can have a reference to an UnsafeCell, without needing memory inside the cell to strictly obey aliasing rules.

I'm not entirely sure if you have to use read/write_volatile, but probably it wouldn't hurt.

1 Like
3

If you go the UnsafeCell route yourself, read the documentation carefully. It doesn't opt out of all aliasing guarantees.

If you have a reference &T, then normally in Rust the compiler performs optimizations based on the knowledge that &T points to immutable data. Mutating that data, for example through an alias or by transmuting a &T into a &mut T, is considered undefined behavior. UnsafeCell<T> opts-out of the immutability guarantee for &T: a shared reference &UnsafeCell<T> may point to data that is being mutated. This is called “interior mutability”.
[...]
Note that only the immutability guarantee for shared references is affected by UnsafeCell. The uniqueness guarantee for mutable references is unaffected. There is no legal way to obtain aliasing &mut, not even with UnsafeCell<T>.

4

Volatile operations order with respect to each other, not with respect to atomics.

Possibly you want atomic volatile, but that is sadly not something rust support, so plain atomic (or adding suitable fences) is the next best thing until that is added.

Plain volatile will work on x86 (assuming the compiler doesn't reorder with respect to the semaphore, which I don't think it should), but I'm not sure it would on platforms with weaker memory models.

5

It doesn't opt out of all aliasing guarantees.

Thanks, I noticed the same thing.

It strikes me that I may have been conflating the rules for mutable references with mutable pointers -- that is, maybe creating multiple mutable pointers to overlapping memory like I've done in the example isn't against the rules. But I think I understand the guidance to wrap shared memory in an UnsafeCell to prevent the compiler from making optimizations based on the assumption that the underlying memory hasn't changed, even if I'm synchronizing access.

6

I'm confused about needing write_volatile. The code looks like this:

//bounce.rs

// Create shared memory with shm_open, ftruncate, etc...
let mut mem = Mmap::options()
    .mode(0o600)
    .create(true)
    .with_capacity("/shared", std::mem::size_of::<Shmbuf<BUF_SIZE>>())?;

let shmbuf = Shmbuf::<BUF_SIZE>::new(&mut mem)?;

// Wait for peer process to write to shared memory.
shmbuf.sem1.wait()?;

shmbuf.buf.make_ascii_uppercase();

// Notify peer process it can access data in shared memory.
shmbuf.sem2.post()?;

And the second process...

//send.rs

// Open already existing shared memory object.
let mut mem = Mmap::options()
    .with_capacity("/shared", std::mem::size_of::<Shmbuf<BUF_SIZE>>())?;

shmbuf.write("hello".as_bytes());

// Notify peer that can access shared memory.
shmbuf.sem1.post()?;

// Wait until peer has modified shared memory.
shmbuf.sem2.wait()?;

let result = String::from_utf8(shmbuf.buf.to_vec()).unwrap();
assert_eq!(result, "HELLO");

Aren't the semaphores enough to prevent reordering?

7

Do you need the shared memory to be written to in a specific order? A normal write can assume you're writing only for the current thread, so the writes could be reordered.

If you call some synchronization primitive, then the compiler will ensure everything is written (in any order) before you call it.

8

Do you need the shared memory to be written in a specific order?

Not during the creation of Shmbuf, which is where I'm calling ptr::write. As long as both semaphores and the buffer are initialized before the call to sem1.wait in bounce.rs, it doesn't matter what order they're initialized in.

But if the part of Shmbuf::new that initialized the semaphores could be moved after the call to sem1.wait, that would be bad.

1 Like
9

If wait() has some side effect or memory barrier, or its body is just invisible to the compiler, then writes won't be moved around it.