when I asked the previous question, I was not sure if it was possible at all
in the past few days, I have spent quite a bit of time playing with AtomicU8 and multi thread updating w/o locks, and threads updating behind each other's backs
I now believe that it should be possible to do mmap in Rust, by using similar techniques as used in (2). The intuition is as follows: In (2), we can have a &[AtomicU8] that multiple threads do read/write w/o locking, and it works. This seems like it should be enough for making mmap work.
Ah, I think, I located the source of the confusion.
If one reads the post title literally (reasonable thing to do), this question does sound quite a bit like the previous question.
However, I don't actually need a literal &[volatile u8]. I only need something that implements the four function interface described above.
I need something where it is possible to read/write a single u8 (even if it is slightly more expensive than a single cpu instruction); and something where we can read/write slices of the mmap relatively efficiently. This is all I need.
Note: I think this solves one of the "changing data behind compiler's back leading to UB" problem for the following reason:
For the data the user of the API can access, nothing changes behind the compiler's back. I.e. for the arguments / return values of the four functions above, they (afaik) obey Rust's assumptions about things not changing.
Thus, as long as we can build these four functions (relatively efficiently) without introducing UB, it should be able to do something "mmap like" in safe Rust. Here "mmap like" is defined as APIs that are okay with just using the four function API above instead of direct access to &[u8].
I hope this clarifies the confusion the title introduced.
The fundamental problem is that other processes are free to open/mmap the file and change its contents, and the OS may update your copy under your feets. This is not guaranteed to be a synchronized operation, so it may always result in a data race, and thus UB, if you're trying to read from it while the OS changes it. AFAIK for the current memory model that's always UB and there's no way to change this. The only solution is having a way to ensure no other process can open/mmap that file.
AtomicU8 guarantees you nothing more that accesses to individual bytes are properly synchronized between different threads. It gives no guarantees about consistency of any larger structures. If you are reading 32 contiguous bytes, the memory could by modified in the middle, so the first and second halves are from different objects.
Of course it's UB. You can't assume that when you read T, you get a valid instance of T. All kind of things will go wrong. For example, you may get an enum with variant tag from one instance of T, and payload data from a different, incompatible variant.
I assume that you end goal is to use mmap as intended. mmap exists so that structures which can be manipulated natively by the language as in-memory objects actually reside on disk, which assumes that things you read should be valid Rust objects. If you are not interested in zero-copy operations and just want to read the raw bytes and then explicitly deserialize instances of T, handling any possibilities of errors and malformed objects --- fine, but at that point what exactly does the mmap give you? You could just be reading a file without any extra whistles and questions about UB.
Your struct should probably also be responsible for unmapping the file when it's done, to make the lifetimes work properly.
You should use raw pointers internally to implement the proposed API.
You will probably be relying on OS guarantees that are more strict than what the Rust Abstract Machine gives you. If that sentence sounds scary, you can fall back to inline asm to make sure you get what the OS is offering.
Thanks. I just now read through that section of Races - The Rustonomicon a few times. Quite useful. I agree with you on #2, I am not 100% convinced on #1 and #3.
I have the following objections:
We are not doing x = x+1, which might compile to
1. r1 = fetch x from mem
2. r1 = r1 + 1
3. store r1 to x loc in mem
If another thread writes to x after 1 & before 3, we can say they access it concurrently. However, in our case, Rust is only doing loads & stores of u8. I believe these compile to single x86_64 instrs. Where is the "concurrent access" ?
What is an unsynchronzied u8 write on x86_64 ?
Sorry if the above sounds pedantic, I'm just not ready to accept this can't be done yet.
Data races are something the language defines abstractly, and how they translate to the hardware is generally irrelevant for whether something is a data race or not. The language does not really define what it means to have a cross-process data race in the first place, so discussing whether one happens is not meaningful unless you are discussing how to change the language guarantees to describe it.
In general, Rust volatile is defined to be used in situations such as when a memory location is treated specially by the CPU so that writing or reading from it is interpreted as an arbitrary IO operation by the CPU (e.g. maybe it turns on an LED). Interacting with memory that changes due to effects from outside the current process seem like it fits that description quite well.