# ABI and MaybeUninit

**URL:** <https://users.rust-lang.org/t/abi-and-maybeuninit/83580>\
**Category:** community\
**Created:** [November 1, 2022, 11:01pm UTC](https://users.rust-lang.org/t/abi-and-maybeuninit/83580 "2022-11-01T23:01:34Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![alice](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/alice/32/9777_2.png) [@alice](https://users.rust-lang.org/u/alice)\
**Post date:** [November 1, 2022, 11:01pm UTC](https://users.rust-lang.org/t/abi-and-maybeuninit/83580/1 "2022-11-01T23:01:34Z")

</div>

In [an existing thread](https://users.rust-lang.org/t/what-counts-as-undefined-behavior/83570/34), @CAD97 makes the following comment:

> [@What counts as undefined behavior](https://users.rust-lang.org/t/what-counts-as-undefined-behavior/83570/34):
>
> > [@What counts as undefined behavior](https://users.rust-lang.org/t/what-counts-as-undefined-behavior/83570/24):
> >
> > It can also store things such as a 2 in a `MaybeUninit<bool>`.
> 
> At the current moment, I'm not convinced that this is the case. Or well, to be clear: you can absolutely _write_ `0x02_u8` to `MaybeUninit<bool>` and then do a typed copy of that `MaybeUninit<bool>`. What I'm unsure of is whether reading that value from the new location is guaranteed to read `0x02`.
> 
> What this comes down to is that `MaybeUninit` is `#[repr(transparent)]` and has the layout _and ABI_ of the wrapped type. This means that with an ABI that for example passes `(u64, u32)` separately in registers, the padding bytes are not preserved on a typed copy, so thus `MaybeUninit<(u64, u32)>` on such an ABI would not preserve the padding bytes.
> 
> This also isn't an issue we can simply just say is a bug in the current definition of the standard library; we absolutely do want to inherit ABI for things like `MaybeUninit<#[repr(simd)] f32x4>` to be passed directly in simd registers rather than general purpose.
> 
> `bool` is probably safe — we say that `bool` happens to match C `_Bool` on all current platforms, but make no promises about other platforms, so we can say that `bool` has the ABI of `uint8_t` C rather than `_Bool` C on the platforms where the ABI of `_Bool` mandates that it only be `0x00` or `0x01`.
> 
> Basically, I'd be careful with `MaybeUninit`. As much as we would like `#[repr(Rust)] union` to be a simple untyped bag of bytes, `MaybeUninit` is by necessity at least a bit more complicated. I'd currently ideally only count on bytes which are potentially valid for the wrapped type being preserved and treat any other bytes as flushing to uninit on typed copy.

Would you be able to elaborate what this means in the Rust abstract machine? Does writing `0x02` to an `MaybeUninit<bool>` just result in an uninitialized value similar to `MaybeUninit::uninit()`, or is there something more we can say?

---

<div class="post-metadata">

**Author:** ![CAD97](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/cad97/32/5076_2.png) [@CAD97](https://users.rust-lang.org/u/CAD97)\
**Post date:** [November 1, 2022, 11:16pm UTC](https://users.rust-lang.org/t/abi-and-maybeuninit/83580/2 "2022-11-01T23:16:37Z")

</div>

So memory itself is untyped, even on the stack. When you write `0x02_u8` to a byte, that byte contains `0x02_u8`, and reading it will produce that value.

Where the potential pitfall is, is when you make a _typed copy_ of that value — that is, you read the memory as a `MaybeUninit<bool>`.

In RalfJ's [MiniRust](https://github.com/RalfJung/minirust) experimental partial formalization, a typed copy consists of essentially decoding the AM-bytes into the abstract value then encoding that abstract value back to AM-bytes at the new location.

So it's a question of defining the decode and encode steps for `bool` and for `MaybeUninit<bool>`.

`bool` is simple enough ([minirust/values.md at master · RalfJung/minirust · GitHub](https://github.com/RalfJung/minirust/blob/master/lang/values.md) pseudo-Rust):

```rust
impl Type {
    fn decode(Type::Bool: Self, bytes: List<AbstractByte>) -> Option<Value> {
        match *bytes {
            [AbstractByte::Init(0, _)] => Value::Bool(false),
            [AbstractByte::Init(1, _)] => Value::Bool(true),
            _ => throw_ub!(),
        }
    }
    fn encode(Type::Bool: Self, val: Value) -> List<AbstractByte> {
        let Value::Bool(b) = val else { unreachable!() };
        [AbstractByte::Init(if b { 1 } else { 0 }, None)]
    }
}

```

The question then is how is `MaybeUninit`'s encode/decode defined, and how does that match with how rustc lowers it to the concrete machine's ABI?

The simple [and desirable](https://github.com/rust-lang/unsafe-code-guidelines/issues/354) definition for `#[repr(Rust)] union` is that encode/decode just copy the AM-bytes directly.

But unfortunately `MaybeUninit<T>` is more complicated, because we want to lower it with the ABI of `T`. The simple answer for decode is then to decode `{ .init: T }`, but if that fails, decode `{ .uninit: () }`. This would then only preserve through a typed copy AM-bytes which are valid for the wrapped type's decode.

And some amount of this may be necessary — at a minimum, internal padding can be nonpreserved at the ABI level, so passing `MaybeUninit<T>` of such a value cannot preserve the padding bytes if it's passed with the ABI of `T`.

The linked discussion is more about `#[repr(Rust)] union` but includes some discussion about `#[repr(transparent)] union` (but more specifically `MaybeUninit`) as well.

---

<div class="post-metadata">

**Author:** ![Aiden2207](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/aiden2207/32/28465_2.png) [@Aiden2207](https://users.rust-lang.org/u/Aiden2207)\
**Post date:** [November 1, 2022, 11:50pm UTC](https://users.rust-lang.org/t/abi-and-maybeuninit/83580/3 "2022-11-01T23:50:00Z")

</div>

At the moment the following code:

```rust
fn main() {
    let two = two();
    println!("{}", unsafe { two.as_ptr().cast::<u8>().read() })
}
use std::mem::MaybeUninit;
fn two() -> MaybeUninit<bool> {
    let mut uninit = MaybeUninit::<bool>::uninit();
    unsafe { uninit.as_mut_ptr().cast::<u8>().write(0x02) };
    uninit
}

```

[Playground](https://play.rust-lang.org/?version=stable&mode=release&edition=2021&gist=4daacd3d8362a12bb6f8871ca76f5496)

passes the miri check and prints the expected value of two. However, outside of a union, I wouldn't recommend relying on this fact. While a `bool` conveniently has the same abi as a `u8`, so thus treating a `MaybeUninit<bool>` as a `MaybeUninit<u8>` is just fine, this isn't true with uninitialized memory in general at all.

In particular, padding bytes are much less safe to move around this way. Even if moving them is well-defined (I'm only mostly convinced this is the case) function calls deinitialize the padding bytes because the behavior of the padding bytes is unspecified in every abi I know of. To treat an arbitrary type as a slab of potentially uninitialized bytes and vice versa, the only safe way to go is using a union:

```rust
union BytesOrT<T>{
     val: T,
     bytes: [u8; std::mem::size_of::<T>()]
}

```

---

<div class="post-metadata">

**Author:** ![CAD97](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/cad97/32/5076_2.png) [@CAD97](https://users.rust-lang.org/u/CAD97)\
**Post date:** [November 2, 2022, 4:09am UTC](https://users.rust-lang.org/t/abi-and-maybeuninit/83580/4 "2022-11-02T04:09:16Z")

</div>

Note that it is [at least somewhat likely](https://github.com/rust-lang/unsafe-code-guidelines/issues/354) that `#[repr(Rust)] union`s will get the "bag of bytes" model where doing a typed copy is defined to preserve all of the abstract bytes as-is. (It's the fact that `MaybeUninit` is ABI-transparent that makes it interesting.)

If they don't get that semantic, and instead have e.g. a bytewise validity union, then that union isn't sufficient to get a byte bag representation, because while it does ensure all initialized bytes must be preserved, it does not necessarily make an uninit byte valid for each byte, nor does it require preserving provenance.

I think it's essentially decided that `MaybeUninit<u8>` is capable of copying any abstract byte while maintaining its full state. As such, if `#[repr(Rust)]` unions don't just natively have the bag-of-bytes representation, I believe the foolproof way of getting that representation is to union with `[MaybeUninit<u8>; size_of<T>()]`.

---

<div class="post-metadata">

**Author:** ![jessa0](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/jessa0/32/19702_2.png) [@jessa0](https://users.rust-lang.org/u/jessa0)\
**Post date:** [November 2, 2022, 4:42am UTC](https://users.rust-lang.org/t/abi-and-maybeuninit/83580/5 "2022-11-02T04:42:59Z")

</div>

> [@Aiden2207](#):
>
> passes the miri check and prints the expected value of two

However, the following program will not, because of the niche-filling optimization ([playground](https://play.rust-lang.org/?version=stable&mode=release&edition=2021&gist=2b42eaa41e43544974f81ee9049e64c1)):

```rust
fn main() {
    let two = two();
    println!("{}", unsafe { two.as_ptr().cast::<u8>().read() })
}
use std::mem::MaybeUninit;
fn two() -> MaybeUninit<bool> {
    let mut uninit = Some(MaybeUninit::<bool>::uninit());
    unsafe { uninit.unwrap().as_mut_ptr().cast::<u8>().write(0x02) };
    uninit.unwrap()
}

```

Edit: my bad, theres a missing `as_mut()` and so the corrected program does actually print `2`.

---

<div class="post-metadata">

**Author:** ![Aiden2207](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/aiden2207/32/28465_2.png) [@Aiden2207](https://users.rust-lang.org/u/Aiden2207)\
**Post date:** [November 2, 2022, 4:57am UTC](https://users.rust-lang.org/t/abi-and-maybeuninit/83580/6 "2022-11-02T04:57:39Z")

</div>

That's actually not why, `MaybeUninit<T>` explicitly forbids niche optimizations. What's actually happening is that `MaybeUninit<T: Copy>` implements `Copy`. With the call to `unwrap` you create a duplicate and mutate that leaving the original unchanged. Adjust your example to operate in place [and everything works just fine:](https://play.rust-lang.org/?version=stable&mode=release&edition=2021&gist=4ddd3bb3e8a7708cce22e875244cc10b)

```rust
fn main() {
    let two = two();
    println!("{}", unsafe { two.as_ptr().cast::<u8>().read() })
}
use std::mem::MaybeUninit;
fn two() -> MaybeUninit<bool> {
    let mut uninit = Some(MaybeUninit::<bool>::uninit());
    // new: vvvvvvvvv
    unsafe { uninit.as_mut().unwrap().as_mut_ptr().cast::<u8>().write(0x02) };
    uninit.unwrap()
}

```

---

<div class="post-metadata">

**Author:** ![Aiden2207](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/aiden2207/32/28465_2.png) [@Aiden2207](https://users.rust-lang.org/u/Aiden2207)\
**Post date:** [November 2, 2022, 5:41am UTC](https://users.rust-lang.org/t/abi-and-maybeuninit/83580/7 "2022-11-02T05:41:13Z")

</div>

The more relevant issue is [this one](https://github.com/rust-lang/unsafe-code-guidelines/issues/73), but you are right, should "bag of bytes" not wind up being the default, the version I originally provided would not necessarily be sound (exact details depend on `T` and what the rules around unions actually wind up being). Using the `[MaybeUninit<u8>; size_of::<T>()]` would be one solution, but an equally valid (and probably more convenient) version would be to additionally add a third zero-sized field:

```rust
union BytesOrT<T>{
     val: T,
     bytes: [u8; std::mem::size_of::<T>()],
     uninit: (),
}

```

This is the same magic that makes `MaybeUninit<T>` work, and does so because all bytes are allowed to be padding (and thus uninitialized) with the ZST in there.

---

<div class="post-metadata">

**Author:** ![CAD97](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/cad97/32/5076_2.png) [@CAD97](https://users.rust-lang.org/u/CAD97)\
**Post date:** [November 2, 2022, 6:35am UTC](https://users.rust-lang.org/t/abi-and-maybeuninit/83580/8 "2022-11-02T06:35:45Z")

</div>

While this is the current implementation of `MaybeUninit` (and I have no reason to believe that it wouldn't continue to work as such), I don't believe that to be as resilient — keep in mind that in this theoretical specification of `union` it doesn't preserve padding bytes, so allowing all bytes to be padding doesn't add any requirements to maintain values through a typed copy.

(Also, `MaybeUninit`'s current definition is, as is core to the discussion, currently `#[repr(transparent)]`, so it could theoretically have meaningfully different semantics w.r.t. what bytes get preserved to `#[repr(Rust)] union`s.)

The rules around what a typed copy of a union actually preserves when it's not bag-of-bytes is extremely subtle. No possible variant has bytes that care about provenance in your version either, so why should a typed copy be preserving provenance?

I both hope and expect that bag-of-bytes (no padding) will be the semantics for `#[repr(Rust)] union` because any\[1\] stronger model seems to me to be much too surprisingly complicated for marginal benefit. `union` just being a way to put names to what essentially behaves as typed views of a named memory region is very simple to teach.

* * *

1. Perhaps until reaching C++'s "active variant" rules... but we have dataful `enum`s already, we don't need `union` to serve that role. The other reasonable choices I know are (medium) to not preserve bytes which are always padding, or (strict) to require all bytes to be valid for at least one variant (all bytes are valid in padding).

---

<div class="post-metadata">

**Author:** ![Aiden2207](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/aiden2207/32/28465_2.png) [@Aiden2207](https://users.rust-lang.org/u/Aiden2207)\
**Post date:** [November 2, 2022, 7:42am UTC](https://users.rust-lang.org/t/abi-and-maybeuninit/83580/9 "2022-11-02T07:42:30Z")

</div>

> [@CAD97](#):
>
> While this is the current implementation of `MaybeUninit` (and I have no reason to believe that it wouldn't continue to work as such), I don't believe that to be as resilient — keep in mind that in this theoretical specification of `union` it doesn't preserve padding bytes, so allowing all bytes to be padding doesn't add any requirements to maintain values through a typed copy.

Sure, padding bytes are not necessarily guaranteed to transfer, but because there's also the byte array in there, a bit-for-bit reproduction is required, because every byte could be padding, or every byte could be meaningful. In other words, every byte is simultaneously padding and meaningful, whereas the array of possibly uninitialized bytes makes every byte meaningful padding.It's a difference without a distinction, other than ease of use.

Additionally, if `T` has meaningful provenance, then a union with `T` as one of the members must also carry that provenance, otherwise, simply writing `T` to the union and immediately reading it back out would be undefined behavior. I don't see where else provenance comes into play, other than the usual caveats involving mixing pointers with byte arrays apply.

Going with the "all unions are a bag of bytes unless otherwise explicitly marked" is definitely the way to go, though. Permitting niche optimizations by default is too error-prone, and if the memory is that important allowing internal tags is likely much more useful than auto-generated niche optimizations anyway. That's not even considering the nonsense that is C++'s active member rules, which, just like type-based pointer aliasing analysis, is a footgunny crutch used so compilers can have decent codegen in a language with very few hard rules. Rust doesn't need any of that because the semantics needed for good codegen are baked into the language itself in a safe and intuitive manner.

---

<div class="post-metadata">

**Author:** ![alice](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/alice/32/9777_2.png) [@alice](https://users.rust-lang.org/u/alice)\
**Post date:** [November 2, 2022, 11:46am UTC](https://users.rust-lang.org/t/abi-and-maybeuninit/83580/10 "2022-11-02T11:46:54Z")

</div>

From this discussion, it sounds like the best way to get a bag of bytes with the size and alignment of a given type is the following?

```rust
union BagOfBytes<T>{
     val: T,
     bytes: [MaybeUninit<u8>; std::mem::size_of::<T>()],
}

```

---

<div class="post-metadata">

**Author:** ![chrefr](https://avatars.discourse-cdn.com/v4/letter/c/e480ec/32.png) [@chrefr](https://users.rust-lang.org/u/chrefr)\
**Post date:** [November 2, 2022, 2:02pm UTC](https://users.rust-lang.org/t/abi-and-maybeuninit/83580/11 "2022-11-02T14:02:04Z")

</div>

I think it needs to be `#[repr(C)]`, [as "Fields might have a non-zero offset"](https://doc.rust-lang.org/reference/items/unions.html#reading-and-writing-union-fields).

---

<div class="post-metadata">

**Author:** ![Aiden2207](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/aiden2207/32/28465_2.png) [@Aiden2207](https://users.rust-lang.org/u/Aiden2207)\
**Post date:** [November 2, 2022, 2:45pm UTC](https://users.rust-lang.org/t/abi-and-maybeuninit/83580/12 "2022-11-02T14:45:50Z")

</div>

No need, scroll up and the reference also says "[the] size of a union is determined by the size of its largest field." That means the largest field must not have a nonzero offset, and in that example, both fields are the largest.

---

<div class="post-metadata">

**Author:** ![system](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/system/32/51372_2.png) [@system](https://users.rust-lang.org/u/system)\
**Post date:** [January 31, 2023, 2:46pm UTC](https://users.rust-lang.org/t/abi-and-maybeuninit/83580/13 "2023-01-31T14:46:22Z")

</div>

Closed on January 31, 2023, 2:46pm UTC

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.