I want to mmap a (potentially large) csv file, and access it like a database. On first open, I write out an index file with metadata which indexes records in the csv file so I can quickly access the correct spot for random accesses. On subsequent updates, the csv file and index are both modified together. If the csv gets modified when my program isn't running, then the index can be rebuilt on next program startup. However, modification by another process while the my program is running is out of scope, and the documentation will explicitly state that doing so will cause problems/data loss.
This is going to be used by several programs I'm writing, so I'll put it in a library crate for my own internal use. The library will mmap the csv file and return &[u8] slices of data from the mmapped region. As I understand it, the library will be safe as long as the underlying files don't change out of process. Should I then mark those methods that return &[u8] as safe or unsafe? (I'm guessing it doesn't matter if I'm just using it internally, but if I were to publish the crate?)
This is a subset of the use-cases discussed in the prior thread.
Note, however, that even if your file is only allowed to be modified while your program is not running, then Rust still must not make any assumption about the 8-bit data it finds. For example, if the index file contains any str, and you later have non-valid UTF-8 characters in there, you quickly end up with undefined behavior, if I understand right. The same holds for bool, etc. Rust relies on the fact that certain bit patterns won't occur in the memory.
You should mark such functions as unsafe, because making them safe would be unsound.
It is one thing that you "guarantee" (by means of having control over your own particular runtime environment) that no other process will change the file. The language specification, however, can't know and doesn't care about the one particular way in which you will run this code on one specific machine. Instead, it requires that programs be correct (free of UB) for all possible environments and executions.
A safe function, by definition, must never, ever cause Undefined Behavior, no matter how hard you twist the environment. If you implement a function that can be called without saying unsafe, then it must not invoke UB even if you change the database file from the outside. Documenting that it will "cause problems" is not enough – it still violates the constraints of the language.
I think I agree too, but I'm not totally sure. But please let me explain. (This is going to be a more philosophical argument.)
Does it really? Doesn't it require that the program will be free of UB within the formal model? I don't think it can require the program to be free of UB in real life.
What if I open my computer and place a 50W radio transmitter near the RAM cartridges. Then this could cause undefined behavior. I know, this is a stupid argument, but my point is: where do we establish a border between things "outside the model" and things "inside the model". Is it an invalid arguement because I open the hardware? What if I stay on the software side and modify the kernel to write into a Rust program's memory, etc.? Or if I start a Rust program with a debugger?
Out of interest, I would like to know if there is really such a formal constraint that we can use to claim that doing mmap is unsafe.[1] I'm not totally sure there is, because it would require that part of the environment to be part of the formal model.
Maybe I'm missing something here, but in eiher way, this is more a theoretical argument. Whether I got a valid point or not, I would still strongly recommend to mark such a function as unsafe.
It means the language guarantee that no bugs/mistakes in safe code can trigger UB condition. If you managed to trigger it in any way only using safe code, it's a bug in some other unsafe code, not in your safe code. And soundness bugs are pretty serious ones.
That definitely is a hardware bug, likely with WONTFIX tag. Hardwares have their own guarantees to allow softwares build abstractions on it. We, as software developers, can't do much if the CPU doesn't guarantee that 1 + 1 yields 2. It doesn't means hardware bugs are impractical - remember those meltdown/spectre crisis? Sometimes softwares try to avoid/mitigate those bugs like we did for meltdown. But usually we just trust them even if we know some bugs unfixed out there like row hammer.
mmap on itself is mostly safe just like any other std::{fs,net} operations. What's unsafe is to get &[u8] or &mut [u8] from memory region that is potentially shared with other processes. You can provide safe API if you don't take references from it or guarantee no other processes can access it using tempfile crate or such.
But in the given examples, you could argue whether it's the "safe code" that triggers UB, or whether it's "the environment" that triggers UB. Only a combination of both will cause UB.
That's why I raised the question whether Rust really guarantees (and can guarantee, or should guarantee) that a Rust program never exhibits UB under any possible environment?
For example, on a platform where I could exclusively lock the file, there could be no safe code alone that's capable of triggering UB, unless something specific happens in "the environment".
To exaggerate:
/// SAFETY: ensure that the environment isn't messed with
/// (no debugging tools, no radio interference, etc. etc.)
unsafe fn main() {
println!("Hi!");
}
Do I really need to guarantee that? Or wouldn't it be sufficient to guarantee that the safe code in my program can't do it on its own?
If you prefer this term, the language defines the border of responsibility. UB is bug no matter what. But safe code do not have responsibility to prevent it. It's duty for unsafe code to prevent UB on any possible safe code.
Memory bugs are one of the hardest bugs to debug so it's important to reduce size of code in question. In practical Rust project only tiny portion of code contains unsafe keyword which are the only code you need to dig into on nasty cases. Normally you write only safe code and doesn't even need to care about UB.
Having shared &mut [u8] or mutated &[u8] is UB. UB is bug no matter what. If you're making library any safe fn must not trigger UB for every possible input. But if it's a binary, there're no such concept of "unsafe binary" in most popular OSes. Generally it's considered OK for binaries to do bad things if user ignores your documentation. Of course it's advisable to do your best to detect it and be helpful instead.
Conflating "X can happen in real life" with "the formal model allows X" is the most common fallacy that makes people misunderstand UB. This is exacly the same situation. If the language specification says that a &[u8] must not have its contents modified by anything else, then it must not, and if you write unsafe code that doesn't forbid/check that, and wrap it in a supposedly "safe" function, then it is unsound.
The hardware analogy isn't relevant because we are talking about software, that's what programming languages are concerned with. A process mutating a file is not an unexpected thing, so other processes must totally anticipate it, while hardware failing is not something that software can realistically help with, so it doesn't need to care inside its own context.
If you think about it: the filesystem is a shared mutable resource, so it's inherently unsafe to assume it never changes. You should think of a file on the filesystem as if it were a global map from paths to contents, except the OS doesn't wrap in a Mutex – which means that you should ensure sharing-xor-mutability. Or, if you like, accessign mmap-ed files must be unsafe for the same reason as static mut is inherently unsafe in this language.
Out of interest, I would like to know if there is really such a formal constraint that we can use to claim that doing mmap is unsafe.
It's the fact that the referent of a &T must never change (if it's not in an interior mutability primitive). I can find exactly where the docs it says that for you if needed, but it's by all means explicitly required.
In this particular case, another part of your program can safely use the fs APIs to write into the mapped file; this will cause UB due to the contents of your mmap region changing while the target of an immutable reference. Therefore, something here has to be marked unsafe.
"All possible environments", if taken literally, would contain environments with broken hardware, and thus is way too strong a statement. More precisely, we want to say "all reasonable environments", but this is somewhat vague.
In practice, we as the Rust community could define this set of reasonable environments to be as large or as small as we want. This isn't something with a single logically correct answer, but rather a tradeoff. I suppose UCG would be in charge of this.
Sorry but is this a joke? I'm not sure I understand the question if it's an honest one.
Ah, I get it now. This code uses safe APIs only to circumvent the type system.
I honestly don't have a good answer to that. I'm inclined to agree with @2e71828 that fs should only allow doing safe things safely. Yes, it may be hard to do in an airtight manner, but after all, there is only a finite number of exceptions, so it should be doable.
In my view, this is a std::fs bug. Accessing /proc/mem and similar should be treated like raw syscalls by Rust, with access forbidden through the safe filesystem APIs. Of course, there should be an unsafe alternative so that we can write safe wrappers that use this class of OS API.
But what if some OS offers other ways to mess up with the system? Should Rust blacklist all of them? I'm not sure if it's really Rust's responsibility.
The filesystem API implementation is necessarily platform-specific already. Part of implementing std on a new platform should be to identify and evaluate the safety of all the platform-provided APIs.
Obviously, this is going to be platform-specific – but I don't see how that is an issue. There are plenty of other ways in which Rust exhibits platform-specific behavior.
It seems like this should just be part of the process for porting the language to a new OS.