There was a discussion about this on internals some time ago. One comment was that there is actually a third dimension to unsafe in current usage, which was "properties that unsafe code can rely on" . Think traits like Send and Sync, or TrustedLenIter.
These require special care since unsafe code is not normally allowed to trust safe code (e.g. relying on Eq to be reflexive and transitive for memory safety is a bug). Currently, they are also annotated using unsafe.
To clarify this, one could speak of "unsafe preconditions" (user must provide these for memory safety) and "unsafe postconditions" (user can rely on these for memory safety). And keep unsafe blocks for manually checked code regions. The idea would be that you need to use an unsafe block in order to do anything potentially unsafe, including calling a function with an unsafe precondition, or returning from a function with an unsafe postcondition. Shorthands would be easy to find: "unsafe pre/post" .
This distinction would solve some longstanding problems with unsafe, for example the fact that you need an unsafe block to implement an unsafe fn (this is not necessary if that unsafe is linked to a precondition: you may not use the precondition in your code). It could also be gradually introduced in existing code, by first making the pre/post annotations optional, then later making them mandatory.
Is this truly necessary at this stage of Rust's evolution, though, considering the amount of churn involved for existing code? I would not know...