I'm doing something weird, probably, but here's some code to illustrate the idea: Rust Playground
What I'm essentially doing there is using something resembling trees that grow, i.e. a enum representing some AST, extensible with different metadata based on a generic parameter tag. I want to disable some enum variants in the Limited version. Any non-disabled constructors have the same metadata between Full and Limited versions (not enforced by the type system, but as a programmer, I guarantee that is the case).
Now the question is, is std::mem::transmute-ing the Limited enum to a Full one "safe"? In other words, does Rust's choice of enum discriminants depend on the data payload of variants? Does it depend on anything at all, or is it just completely arbitrary?
Now, if that enum didn't have #[repr(u8)] annotation that would obviously be unsafe (furthermore, Rust would refuse to transmute differently-sized types). With #[repr(u8)] the layout is guaranteed to be the same, and it seems to work in the limited testing I did. But if Rust can arbitrarily assign different discriminants to the "same" enum variant between A<Limited> and A<Full>, this would be woefully unsafe.
For context, the actual enum is significantly more involved, including recursion, hence I want to avoid traversing it.
Apologies if this was asked before, this is kinda niche and specific, so searching for it turns out to be very hard.
unsafe is not for arbitrarily breaking the high-level type system. It's not to be used lightly. It's for when low-level, primitive operations can't be proven by the compiler to be safe (e.g., FFI with C code that doesn't have static ownership/borrowing information, or doing very simple operations over built-in types, like implementing integer to byte array conversions).
If you share the actual problem domain with us, we can perhaps help you write performant code that doesn't need to be wildly unsafe.
By the way, this looks even more dubious than your original example – not only does it transmute a single value, it also transmutes every pointed value, transitively, including pointers — transmuting of which is almost always wrong. Given a pointer-like type P<T> of pointed type T, it is not guaranteed that P<T> has the same representation as P<U>, thus your transmute with the recursive Box<Self> may well be UB.
AFAICT, it isn't, at least with Box, unless the initial example is UB: both pointed-to types have the same size and alignment, and pointed-to value is valid as both types as long as discriminants are the same. But as I said, I don't want to sidetrack the discussion.
Ugh, I shouldn't have used (). Assume for the sake of argument I've ensured size is indeed the same (union is as large as the largest type in the union, so worst case padding with a bogus variant is an option)
Edit: actually, just simply adding align(xx) on the enum would serve the purpose, where xx is the size of the largest removed variant (rounded up to a power of 2).
I haven't looked at exactly what you're trying to do in depth, but a prerequisite for it being sound would be: Is the layout of enums guaranteed? They are not guaranteed in the default representation, but you can guide the representation.
Thanks for the link. I've seen that page, and it is indeed quite handy. Unfortunately, it's not very clear from it whether data-carrying enums follow the same rules as fieldless. The Rust reference, as pointed out by @2e71828 above, is a somewhat more explicit about it.
This is not relevant to the original question, but no, they are not equivalent. The text describing primitive representations says
The representation is … a repr(C) union of repr(C) structs for each variant with a field.
That is, #[repr(u8)] enum is laid out as if you wrote a certain #[repr(C)] union, not as if you wrote #[repr(C, u8)] enum.
(The difference between repr(u8) and repr(C), given how each is defined, is that repr(C) has a consistent amount of padding between the discriminant and the field data — the data always starts at the same offset regardless of which variant you are looking at — and repr(u8) does not have that consistency. So, repr(u8) may be more compact but also may be harder to use for FFI.)