One of the cases that has been brought forward for why we should require &T to point to a valid T is that people want &! to be uninhabited so that functions taking &! can be just removed from the binary.
Also, you seem to assume that a &[u8] can point to any data. That is far from clear. The relevant discussion for that is here, but the summary is that we might want to require u8 to be initialized, which means that e.g. the 2nd byte of a (u8, u16) (which is padding, assuming it gets laid out in the order I wrote it) is not a valid u8.
At this point I have to insert the mandatory warning that mem-mapping interacts very badly with Rust's reference guarantees. See this discussion for further details.
Congrats, you managed to hit 3 edge-cases at once. 