What's the current state of soundly and efficiently turning a &mut [MaybeUninit<u8>] into &mut [u8] filled with garbage? (u8 here stands in for any type for which all bit patterns are valid.)
As I understand it, this should be possible by writing 1 byte to each memory page spanned by the slice so that the pages are actually mapped as a kernel matter (in case safe code chooses to read from the slice before writing to it) and telling LLVM to consider a freeze over the slice memory to have taken place.
For a long time, I was told that the Rust side was waiting for LLVM to get "freeze", but lately I've been told that the LLVM side is already done.
Is there a library function for performing this operation?
I can't see a direct reference to those being a reason for Rust avoiding adding freeze, so there could be more causes. Here's a (fairly confusing) thread a few years ago on internals
That does mention LLVM freeze may or may not actually have anything to do with unaccesed memory?
Note that freeze, if ever implemented, will likely work on values rather than places. That means you won't be able to convert &mut [MaybeUninit<u8>] to &mut [u8] with it anyway.
The use case is preparing an output buffer such that:
The overhead of actually writing zeros up front to every element of the buffer is avoided, since such overhead is pointless, when those elements are about to be overwritten.
The buffer has the ergonomics and API compatibility of &mut [u8] instead of MaybeUninit<u8> contaminating various layers.
Without unsafe audit debates about what is documented to be UB vs. what compiler internals actually treat as UB.
Some things you should keep in mind when considering the value of this feature are that:
Obtaining zeroed memory may not have as much overhead as you think. It’s such a common requirement that OS memory management may have special cases to support it.
Using uninitialized memory means that if there is a bug in the application, the consequences of the bug are not just extraneous zeroes but potentially leaking information that should be secret.
They were probably referring to the special-casing of vec![0, n] which gets optimized to __rust_alloc_zeroed.
I never really understood this argument. Yes, secrets can be leaked by misusing APIs, but that is true for safe APIs too. Think of the various ways in which devs casually "recycle" Vecs, for example, by writing to indices of v.as_mut_slice() rather than truncating and then pushing to it.
Commonly used counter-argument against "freezing" references is the existence of MADV_FREE (I don't like it either, but it exists and has to be accounted for). Imagine you allocated [MaybeUninit<u8>] on heap, the allocator may give you memory marked with MADV_FREE, so before you write something into it the memory may transform from "garbage" to zeros at any moment. In other words, reading twice from the same exclusively referenced memory may produce different results, which breaks the compiler assumptions about how memory works, i.e. it's an example of UB.
So at the very least the reference freezing operation would need to write at least one byte into each referenced page.
If by “adding initialized elements to the end” you mean using Vec::push() or Vec::extend(), then: no, those operations will overwrite the uninitialized memory (within the spare capacity) with the provided values; there is no extra write of zeroes.
But your original question, as I understood it, was about obtaining a [u8] to write into, not a Vec to write into. These are different cases. Vecs do work with memory that is left uninitialized until it is to be used.
Please don't do that. The "garbage" is the HeartBleed vulnerability and ASLR bypass. Letting leaked sensitive info spread through safe references undermines the safe/unsafe boundary.
I can help you fix buffer handling debt in encoding_rs to use write-only/append-only abstractions that don't rely on &mut [u8], but you need to attend to the project more often than once every few years.
Doesn't compiler elide that overhead (in your case)? If not, perhaps your code can panic with part of buffer not written.
BorrowedBuf in std::io - Rust is useful. (Also, as it's a nightly feature, it collects feedback on its API; if something is missing you can request that it's added.)
@hsivonen Hi! Note that the rustix crates offers rough approximations of std::io::Read::read_buf and BorrowedBuf that you can use in stable Rust right now.
I addressed this in the post starting this thread.
It seems that there isn't a library funtion that already does this. Writing a zero byte into each memory page spanned by a &mut [MaybeUninit<u8>] is easy enough to do and, as I understand it, should make subsequent two read instructions from a given non-zerod location in the slice return the same value for both reads.
Clearly, the compiler has the capability of assuming that the contents of a &mut [u8] are unknown but stable: That assumption has to be made for a &mut [u8] that has come from far enough that the optimizer can't see where it came from.
When the optimizer can see that I have &mut [MaybeUninit<u8>] and I have taken the above-described steps to make things stable as far as the kernel and ISA are concerned, what's the documented-to-be-correct compiler-level way to interpret the &mut [MaybeUninit<u8>] as a &mut [u8] so that the compiler will treat it as if it came from far enough away for the optimizer not to see where it came from?
There are multiple comments whose general sentiment is that I should not want to do this and also multiple comments suggesting an abstraction struct that wraps the non-yet-initialized spare capacity.
I agree that I should not want to do this, but the thing I actually want requires an addition to the C++ memory model, so to write Rust code today, I know that what I really want isn't available, which is why I'm asking what I'm asking.
What I really want is a new language-level notion of write-only references. That is, I want &sink [u8] that would borrow check like &mut [u8] and would have all the write capabilities both in the sense of Rust operations available to the programmer and write optimizations available to the compiler (reordering, widening, dead-store elimination) and ergonomics of &mut [u8] but it would not allow either the programmer or the compiler to cause reads from the referenced memory.
This would be useful not only for not having to zero-intialize buffers before writing to them with all facities presented by slices and core::simd but would address the issue of writing to an untrusted buffer in scenarios like Wasm runtime writing to a SharedArrayBuffer, hypervisor writing to guest-provided memory, or an OS kernel writing to a userland-provided buffer.
It's bad that Rust, C++, and C are positioned as the languages for writing kernels, hypervisors, and VMs, but the memory model doesn't accommodate this case. (Neither atomic nor volatile are appropriate: volatile inhibits too many optimizations. Atomic not only inhibits optimizations that would be fine but is also defined not to be UB only if the other side adheres to the way atomics are implemented, so atomics aren't OK when the other side should be assumed potentially hostile.)
Thank you. I might end up taking you up on that, but, perhaps foolishly, I'm still trying to find a way to make language-level slices work first. I think it's bad to have to create an abstraction for a basic case like this instead of being able to use what slices already provide.
I'm almost done re-achieving the perf of the main branch of encoding_rs on Zen 3 using safe code as seen on the unroll branch. That branch combines features provided by slices, arrays, and core::simd in various ways, and I think it's bad to accept to have to re-create all that standard-library API surface outside the standard library to create a write-only slice when the standard library doesn't provide one.
That's why I'm pursuing using a regular mutable slice, refraining from reading from it, but doing the minimum to satisfy audits so that even if there was a read, it wouldn't be UB. (I'm saying "audits", because my understanding of what Gankra and Ralf Jung have said is that the current docs over-claim UB relative to what the compiler is currently internally trating as UB in the relevant case.)
I'm sorry that my misestimation of the prioritization situation that I'm dealing with has resulted in your contributions remaining unpublished to crates.io. Updates to the crate cause so much bot and human update churn that I don't want to do two releases if I think I can get another thing in the same release soon. But so far, I have been way too optimistic of how near getting also the next item in the same release has been, so "soon" hasn't been at all soon.
The definition of the Rust type [u8] requires it's initialized, so you're at least talking about [MaybeUninit<u8>], which of course can be uninitialized but must be unsafely asserted by you to have been initialized.
In you're talking at the LLVM level, it's a fairly similar situation as I understand it: it's illegal to assign a poison or undef value to a type without those annotations: that is, you will cause UB if you do so. The freeze instruction act as a barrier to optimizations that prevents these UB hazards from getting propagated through eg inlining and constant folding. Note that there's no actual "make the value defined" step there, just avoiding making assumptions.
As far as exposing that in Rust, you could just use assembly, I guess? But I'm quite sure if it was that easy, it would already exist.
This can be supported in a library (potentially std) too. BorrowedBuf does something like that.
Just that type existing however likely does nothing for you. You might want to use it with e.g. Read and Write, so those will have to support this new type too, and that's not easy nor quick to do.
The biggest problem with this approach is how to track which part of the slice has been written.
Assuming that you rule out MADV_FREE pages by writing a byte to each page or otherwise knowing where the memory came from, this code has the effect you want:
unsafe { asm!("/* {0} */", in(reg) ptr) };
After this unsafe block, you can treat the memory as initialized.
This follows the guidelines explained by How to use storytelling to fit inline assembly into Rust with the story of the inline asm block being a loop that fills the array with bytes, thus initializing it. The bytes written are chosen arbitrarily, which is why an empty asm block is a correct implementation of this story (assuming no MADV_FREE involvement).
Note that this is not equivalent to the LLVM freeze operation because it uses a story that also overwrites bytes that are already initialized with potentially different values, whereas a real freeze operation would guarantee that any bytes that are already initialized do not change.
If you want it to be compatible with miri, then the standard way to implement that is a cfg where you execute the story you chose under miri, and otherwise run the asm block in a normal execution.
if cfg!(miri) {
for ptr in slice::from_raw_parts_mut(ptr, len) {
*ptr = 0;
}
} else {
unsafe { asm!("/* {0} */", in(reg) ptr) };
}
If ptr points to an allocation of len bytes containing uninitialised memory, then wouldn't *ptr = 0 cause UB, since assignment calls drop on previous value? I thought that initialising memory has be done using std::ptr::write (or ptr::write method).