Size is not a property of a type, it is a property of its representation. And compiler is not required to choose the smallest representation possible.
So even if 0 is way too much to represent all possible values of !, compiler is still allowed to choose such a wasteful representation. (Some reasons why it might want to were already stated.)
This does imply an infinite amount of padding bytes, which does mean you can't eg. memcpy this uninhabited type 
We could have &! be a pointer-sized uninhabited type.
(I'm of opinion that having a value of &! should be instant UB, too, since compiler could insert a spurious read of it at any time.)
But all of them are contained within zero-sized allocation ... right? Where else would they be?
To the left, obviously!
Going on a spin-off, it would be nice to split zero off into its own variant since ZSTs require very special handling in many places.
enum AllocationSize {
MinusInfinity,
Zero,
Positive( pattern_type!(usize, 1..=(isize::MAX as usize)) )
}
Going on another spin-off, which types can even be inhabited with type argument of !? A surprising lot of them, from &[!] and PhantomData<!> to fn(!) -> bool.
This made me appreciate the separation between value and place more thoroughly. Thank you!
To @jthulhu, while the type Infallible may have no values, the language still allows programmers to describe places, such as variables or struct fields, to which all none values may be assigned. References don't refer to values directly; they do so through the place those values occupy. There may be multiple distinct and distinguishable places where an Infallible could be stored, even if no value can be ascribed to those places because Infallible is uninhabited.