I've seen examples of using shared memory for communicating between threads, but not between processes.
What needs to be done to prevent UB if a Rust process needs to both produce and consume values in shared memory? Is using atomics necessary or sufficient? Do I need to use UnsafeCell?
It likely depends a lot on specifically how (by what mechanism) you obtained the shared memory. Are we talking about memory mapping? (If so, that's notoriously difficult to get right.)
In any case, some sort of interior mutability and unsafe is likely going to be needed in any case, since if you have memory that some other process can modify, then Rust can't prove that it obeys the shared-xor-mutable rule.
You need some kind of synchronization to safely communicate using shared memory between processes without data races, just the same as between threads (since communicating between processes is also communicating between threads by definition). You can't assume that any regular kind of mutex will work though; the requirements will be OS-dependent. pthreads mutexes are safe to use with shared memory as long as you can ensure that one process always initializes the mutex before the other attempts to use it (initialization is not threadsafe), but Rust no longer uses these as the implementation of its default mutexes, and I don't know what promises about shared memory the current implementation has (if any).
You can do your own synchronization using atomics but this comes with all the normal caveats about this being tricky to get right if you don't know exactly what you're doing.
And yes, for Rust in particular some kind of interior mutability will have to be used to enforce the borrow checking rules.
Previously on linux std::sync::Mutex used boxed pthread mutex since the pthread API requires stable address even between move. Since the mutex is placed in process heap it couldn't be used within shmem. But now it uses inline futex so it's OK now.
The windows impl definitely isn't interprocess because it uses a SRW lock which is documented as not supporting interprocess locking, so it definitely wouldn't be cross platform if you used the std Mutex.
You can use the pthread_mutex* types and functions in libc to create mutexes that can be shared between processes. Youll have to configure the attributes correctly for shared mutexes though
Oops i don't think that applies to windows. I thought the libc crate annotated when functions were only available on certain platforms but that doesn't appear to be the case.