Is zeroing a repr(Rust) enum UB?

The zeroize crate zeroes an Option<T> given a mutable reference to it by zeroing out its backing memory. This raised red flags for me, as I believe the memory layout and set of valid bit patterns of Option<T> is only defined for certain T specified in the core::option module-level documentation. I believe as there is a live mutable reference to the Option<T> at the time, setting it to an all-zeroes bit pattern may be instant language-level UB.

However, I couldn't get any segfault or weird behaviour to happen in zeroize by playing with nested enums, bools, and NonZero*. It seems all-zeroes is currently always chosen as a valid bit-pattern in the cases I tried, though I don't think it's guaranteed to be that way in the future. The closest I could find was an enum containing an uninhabited member in which you can't safely construct an all-zeroes bit pattern, but using core::mem::zeroed still seems to work ok: Rust Playground.

Correct.

The current implementation does more optimizations, but they should not be relied upon.

Here's a quick UB example:

#[derive(Debug)]
enum Foo { A = 1, B = 2 }

fn main() {
    let x: Option<Foo> = unsafe { std::mem::transmute(0_u8) };
    dbg!(x);
}

https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=a86270a4f0d74f55d1655a1d4a984f2b

Error from miri:

error: Undefined Behavior: type validation failed at .<enum-variant(Some)>.0.<enum-tag>: encountered 0x00, but expected a valid enum tag
 --> src/main.rs:5:35
  |
5 |     let x: Option<Foo> = unsafe { std::mem::transmute(0_u8) };
  |                                   ^^^^^^^^^^^^^^^^^^^^^^^^^ type validation failed at .<enum-variant(Some)>.0.<enum-tag>: encountered 0x00, but expected a valid enum tag
  |
  = help: this indicates a bug in the program: it performed an invalid operation, and caused Undefined Behavior
  = help: see https://doc.rust-lang.org/nightly/reference/behavior-considered-undefined.html for further information
1 Like

Imagine GreaterThanOneUsize which would have 0 and 1 as a niche. If Option picks 1 to represent None, then observing a 0 is UB.

Here are some examples where this break things:

  • Using Result and actually causing a segfault Rust Playground

  • Using Option and breaking the invariants of a type (which however aren't encoded in the language/compiler) Rust Playground

1 Like

Thank y'all for the examples. Let me be more specific since we've deviated from the original case which is a bit more ambiguous.

The code in question is generic over Option<T> for essentially any T. It takes a &mut Option<T>, drops any value inside using Option::take, zeroes the backing memory, and then sets the backing memory back to None using ptr::write_volatile. The place is never read from while it had a potentially invalid bit-pattern.

The Rust language reference actually doesn't seem to list this case as UB, though I know there's been lots of discussion about expanding that list of UB. The only mention of enum discriminants only seems to apply if the place is assigned to or read from through a reference, if I'm reading this correctly: Behavior Considered Undefined.

It says that producing an invalid value, even in private fields and locals is UB and that "Producing" a value happens any time a value is assigned to a place. That sounds like it matches your situation.

2 Likes

I think the term might be used wrongly. To my understanding, assignment implies replacing the old value with the new value and dropping the old value. A let statement does not qualify as an assignment and neither does ptr::write nor mem::zeroed. All of them write a value to a memory location, but they do not read the previous memory to drop a value.

Either the documentation is using wrong terminology, my understanding of the terminology is wrong or a simple write is indeed not UB.

I was interpreting writing through a pointer cast to *mut u8 as not "assigning to a place". If there is no outstanding reference to a place, and only pointers, is it still a place?

I would qualify let var = value; as an assignment to a place, as the let creates a new place and the assignment writes a new value there.

I always visualized an assignment as drop(std::ptr::replace(*mut location, value)).

The distinction I'd place here is assignment drops the previous value before memmoveing the new value there.

This is why, for example, MaybeUninit::write exists as distinct from *m.assume_init_mut() = foo; -- the former overwrites without dropping what's there, whereas the latter is an assignment.

This specific case, as you've described it, is (probably) sound. Converted back into code:

fn zeroize<T>(r: &mut Option<T>) {
    let p = r as *mut Option<T> as *mut [u8; mem::size_of::<Option<T>>];
    unsafe {
        ptr::drop_in_place(p as *mut Option<T>);
        ptr::write_volatile(p, mem::zeroed());
        ptr::write_volatile(p as *mut Option<T>, None);
    }
}

Memory in Rust is untyped. In memory, a value is just a bag of bytes[1]. The "only" time a type's backing memory is asserted to be valid is any time a reference is used[2] or a "typed copy[3]" is performed.

So all we're doing here is converting the input reference to a raw pointer, then doing three things:

  • Dropping the typed value;
  • Writing all zeroes to the memory location behind the pointer, in the size of the owned memory; and
  • Writing the typed value None back.

All three operations are sound, and we exit the function with the world in a valid state for the reactivated reference, so the combination of these operations is sound.

Note that this would be unsound if we used r directly instead of p, as r would continue to reassert the validity of the backing memory. It is only through the raw pointer that we get access to the raw untyped memory.

There is one remaining pitfall, however: this implementation is unsound if dropping the value unwinds. This will result in the calling frame still having a reference to dropped memory (and eventually, the owner dropping it again). This can be avoided with a simple unwind-to-abort guard.


[1]: but bytes are also complicated; What's in a Byte?

[2] "used" is used here very informally; even just mentioning the name of the binding is potentially enough to count as a use. Reborrowing definitely is, and references are transparently reborrowed quite often.

[3] basically, moving of the value into a named binding with a type.

1 Like

That's a great reduction / analysis, thank you.

I do wonder whether optimization, esp. inlining and code reordering, could affect whether this is valid or not. I'm not clear on the guarantees of volatile writes, but if a line of code referencing the Option<T> were moved after the ptr::write_volatile(p, mem::zeroed());, havoc could ensue. Ralf Jung also seems to argue, though the situation Ralf describes is more straightforward, that we shouldn't try to write exceptions to the "producing invalid values is UB" rule because it'd be too complex.

Although isn't merely creating and using (moving) the result of mem::zeroed() UB immediately? After all, it constructs a typed value with an invalid memory representation (if 0 happens not to be the representation of None).

1 Like

Right. But in this case, zeroize actually casts to *mut u8 and sets the bytes to zero (effectively a memset).

I get that, but the problem is not with the raw pointer or with the validity of the write. The problem is that the zeroed() call itself creates a temporary value with an invalid bit pattern. This in itself is UB — even without the pointer cast or the write_volatile() — if the official documentation is to be believed.

Oh, what I'm saying is zeroize doesn't actually use zeroed(), that's something rather only in @CAD97`s code

Alright, that makes sense.

I found this discussion in the Unsafe Code Guidelines WG which seems to match the scenario in zeroize. It seems to not have been decided as of yet.

The type for this copy (and value) is [u8; mem::size_of<Option<T>>()], not Option<T>. All-zeroes is a valid byte pattern for [u8; N].

3 Likes