Atomics in shared memory

Hello everyone!

Recently, I was experimenting with shared (memory-mapped) memory between processes, and synchronization between them. On Linux, there are POSIX semaphores that can be placed in shared memory and be used for synchronization, so I guess as long as all access is volatile and synchronized, everything is fine. I wonder about atomics in shared memory -- Rust standard library atomics have a from_ptr() unsafe constructors that take an aligned pointer and give an atomic object reference to perform operations -- however, I'm currently not sure whether it's okay (for example), for &AtomicI32 to point to a shared memory. Standard library atomics are implemented with UnsafeCell, which allows the value to be mutated while there are shared references to the cell (but not mutable references to the cell or shared references to the value itself), however, I've read that Rust references are "non-volatile" (meaning the compiler assumes that the value cannot be accessed/modified "from the outside") -- does that apply to UnsafeCell references? If a value can be safely modified from other threads (provided that a modification uses atomic operations only), can similar modifications safely happen from other processes (or signal handlers)? Or is it better to use atomic intrinsics for shared memory instead?

1 Like

I can’t comment on shared memory, but I can say that UnsafeCell opts out of this rule for the data within it — otherwise atomic types and Cell wouldn't be possible to implement. However, do note that this only applies to shared references; as soon as you take an &mut exclusive reference, even to an UnsafeCell, there must be no other read or write to the memory.

One potential issue as far as I understand is that the lowering of atomics to machine code might not be the same between different programming languages / compilers. That is, there are multiple mutually incompatible ways to lower the atomic operations. And you have to choose one set and stick to it (for any given atomic at least). As a concrete example, it could be that in one lowering the memory fence is on the write side, but in another lowering it the memory fence is on the read side.

Famously, the Linux kernel uses it's own LKMM memory model that predates the now common C++11 model used by C++, C and Rust. A side effect of this is that the lowering in that case is different. But even if you start out with the same high level model, there may be different lowerings to choose from.

That doesn't really answer your question, just point out that there are many things to keep in mind between processes that are not built by the same compiler (or even compiler version).

And for example, this is leveraged to allow Atomic<T>::get_mut(&mut self) -> &mut T.

Unfortunately the get_mut method would then be unsafe, since the compiler can't guarantee that only one process has a &mut reference to the atomic. I assume if you don't call this method you're Ok, but it does at least create a risk.

Rust follows the C/C++ rules for atomics as documented here. So the best way to determine whether they work (apart from the issue above) is probably to find documentation on how atomics work with multiple processes in C/C++.

get_mut() requires a mutable reference to the atomic, while from_ptr() gives only a shared reference

2 Likes

Those two methods have different purposes. You may be thinking of from_mut. get_mut allows exclusive access to an existing atomic (without an atomic operation) after it has been created. I was just pointing out that it cannot be called safely in the case of shared memory among multiple processes.

I don't understand your comment? OP never mentioned from_mut or get_mut, they talked about from_ptr from the start. It was you who mentioned get_mut/from_mut.

And since from_ptr returns a shared reference, you cannot call get_mut on it.

Some relevant UCG threads:

Though there are likely a few more issues in that repo that are relevant.

1 Like

You're right, I didn't realize that from_ptr returns a reference, but of course it has to. Therefore get_mut is not an issue.

As far as I understand now, the "non-volatile references" problem was that the compiler was allowed to insert spurious reads on memory pointed to by a reference, which may be undesirable for device-based MMIO. For atomics shared between processes, this doesn't seem to be a problem (at least not more than it is for atomics shared between threads of the same process).

1 Like