Thanks for everyone's responses. There is a lot of great information here, but it seems like reading uninitialized memory will always be UB. The closest I can get is to read effectively uninitialized memory returned from system calls.
I was sort of hoping that something like LLVM's freeze instruction could be exposed, but I couldn't find any work to make that happen.
There have been some discussions about freeze, though no real proposals yet for how to integrate it into Rust:
There are plenty of ways that page can be full of random byte values that sum to zero.
Calling assume_init on data that has not been initialized is the source of UB here. So if you assume that that line will do something like "follow this pointer, add a bunch of things and compare their sum against zero" you are already mistaken. page.assume_init() is, at least conceptually, instant UB by itself.
Naturally, the compiler will almost certainly emit some code here that does do something like that. But even if it does, "panicking" is also not UB, so the result of the comparison still has no bearing on whether the code as a whole is UB or not.
Note that freeze still doesn't solve your problem. It just replaces your UB with an arbitrary value. So if the page is not initialized, you're still just testing if arbitrary_compiler_chosen_value() == 0, which doesn't give you any usable information -- it could still look like it's zero-initialized even though it's uninit.
As long as I can read uninitialized memory everything else is a detail. For example, I could be testing a memory allocator that is supposed to reuse memory pages after zeroing them; so I would allocate, check page, modify, free a bunch of times. But I can't do that if even looking at the page before writing is UB. Freeze would solve my problem because it allows me to write this code without modifying the allocated page.
That only works so long as it doesn't return uninitialized pages. If there's a bug that returns them uninit, then the code you write using freeze could say they they have the expected value because the compiler optimized out the check.
freeze just removes the UB; it doesn't let you actually "read" it.
BTW what about the following code? IIUC it does count as UB and compiler is not allowed to replace reads from the resulting page with something arbitrary
extern "C" {
// does nothing with data behind the pointer,
// but works as an "optimization barrier"
fn touch_pointer(p: *mut [u8; 4096]);
}
let mut page = Box::<[u8; 4096]>::new_uninit();
let page = unsafe {
touch_pointer(page.as_mut_ptr());
page.assume_init()
};
// we now can legally read data from `page` since compiler
// can not make any assumptions about `touch_pointer`
That code has UB, but UB allows anything to happen, and "does what I wanted" is included in "anything". You should just be aware that changes to the behavior are not considered breaking even if they break your code with UB.
Why would it not allow me to read it? freeze says that the page will have an arbitrary but unknown pattern, which would mean that the compiler can't remove checks based upon knowledge of the value.
I really want to avoid solutions that rely on the compiler being too dumb, cause eventually it will figure it out. Sort of the opposite of the "sufficiently smart compiler".
Assuming touch_pointer comes from a shared library, compiler in principle is not allowed to make any assumptions about it and peeking into its implementation. But yes, it may break with static linking and LTO.
I think the point that @scottmcm is trying to make is that the compiler may optimize the code below to always print hello world without actually reading the underlying value.
let mut uninit = MaybeUninit::<u8>::uninit();
let ptr: *mut u8 = uninit.as_mut_ptr();
freeze(ptr);
if *ptr == 0 {
println!("hello world!");
}
This is because as described in the LLVM reference, the resulting value must be fixed, but can be arbitrary. The value 0 for the memory location is valid under those constraints.
According to my reading of the reference, it should be a fixed but unknown value. The "arbitrary" part refers to fact that it could be anything, so the compiler can't optimize it by assuming that it is some particular value. The examples further support this, %z = add i32 %x, %x being even but %cmp = icmp eq i32 %x, %x2 being true or false with the compiler not being able to tell.
Of course, I am a novice and this is a really difficult subject so I would appreciate any other references supporting or refuting this stance.
The problem here is that if the compiler is ever smart enough to know that the memory is undef, then there's nothing you can do to detect that. At best you can use freeze to make your checks useless instead of UB.
All the approaches of things like the "optimization barriers" or "would mean that the compiler can't remove checks" are about forcing the compiler to be insufficiently smart to be able to actually know anything useful -- which makes those things also "work" fine without the freeze.
Make sure to go through https://llvm.org/docs/LangRef.html#undefined-values really carefully. In particular, note the difference between freeze(under) and just undef.
If you have %A = undef and %B = freeze(undef), icmp %A, %A is undef but icmp %B, %B is i1 1. It's that repeated use where the freeze is relevant. If the only use of %B is in icmp %B, 0, then it can be optimized to just freeze(undef) -- it'll have a consistent answer, but the optimizer can pick whichever answer it would prefer.
(At least, that's my understanding. I'm no Ralf, though 
)
I would generally consider the word arbitrary to refer to the guarantees that are given to us, the authors of the code, and not to be about what the compiler itself is allowed to know.
I'm no expert in the LLVM reference, but this is how we use the word "arbitrary" in pure mathematics, and I assume it is used in the same way here.
In general, I don't think the LLVM reference ever says anything like "the compiler is not allowed to know X", and I think it would be bad design for a language reference to say something like that. There are always much better ways of writing this in the reference, e.g. "it is guaranteed that the frozen value is the value actually stored in RAM at that location" or something like that.
I think that this is actually an interesting case that deserves a little more examination.
As written, if we take your comment on the touch_pointer function to be correct, the code does have undefined behavior (as stated above by others).
What would happen if I replaced the C of touch_pointer with a function that did, in fact, initialize the 4096 bytes of memory pointed to? The memory would then be initialized, and the program would execute without UB.
You can't decide whether that snippet has UB or not without checking the actual definition of touch_pointer. This kind of whole program analysis is one of the major sticking points of C code that Rust set out to prevent. (Which is why you need unsafe to bring in this ambiguity.)
Another way of saying the same thing. The compiler can't decide whether that snippet has UB or not without checking the actual definition of touch_pointer. The "UB bomb" will "explode" when the veil of ignorance is stripped away.
In my opinion the snippet does not contain UB and it should not change in the presence of static linking and LTO. But I agree that it may.
UB is a matter of language specification and since Rust does not have a proper specification the answer can go both ways. In my opinion, it's reasonable to treat extern functions as "unoptimizable" in the spec, no matter how it gets linked and whether LTO is enabled. Same should be applied to pages which are returned by memmap and other similar syscalls. I don't think that they should be included into language specification (as memcpy). If compiler does not know anything about their semantics, then it has no choice but to treat them as any other unoptimazible extern fn.
Note that treating "raw" syscalls in such fashion does not prevent us from adding to the spec their aliases, which will have semantic meaning for compiler and thus could provide additional optimization opportunities, e.g. marking page memory returned by a hypothetical memmap alias with undef and marking it "pure", so its call may be eliminated if a returned page is not used anywhere.
What does that mean in the spec, though?
While this may be true, remember that the language model is operationable (defined by operations over state) and the as-if rule, not by some specification of what optimizations are allowed to do.
So sure, calling an extern fn that may potentially initialize the memory must behave properly in the case that the memory is initialized. But remember: MADV_FREE means that uninitialized memory can be non-deterministic in practice, not just poison in the abstract machine.
The manifestation of UB in this case would be because when you use the value, the compiler assumes that it's a normal value. But if it's uninitialized in a MADV_FREE page, two spurious reads may return different values the compiler assumes are equivalent, and you have UB manifesting because of it.
Is this manifestation rare and contrived? Kind of. Will it work fine most of the time? Sure. But importantly, it's still UB and it will still misbehave, and an optimization barrier doesn't prevent the UB from manifesting.
The only way to prevent miscompilations/misbehavior is to eliminate the UB of using an undefined value by freezeing the value to a non-deterministic but consistent non-undef memory value.