That is a very good point. Pretty much any safe code can be used to cause UB in the presence of unsafe code.
But subtraction is an operation that is common and useful even without the presence of unsafe, whereas pointer casts are pretty exclusively used in combination with unsafe code.
As a related example raw pointers could safely be Send and Sync, but they aren't, because types that contain them very likely wouldn't be safely Send and Sync, when the pointers are used.
When the aren't used, any type that contains raw pointers could safely be Send and Sync.
The following code is completely sound and the unsafe traitimpls in it could theoretically be unnecessary:
Raw pointers are not Send and Sync, not because its inherently dangerous, but because there is a high chance of it being wrong (but that'll only happen in the presence of unsafe code).
I'm not trying to argue that pointer casts should be made unsafe, but I think there are some similarities between these situations.
I do agree that as is a bit too undifferentiated for how short it is. It ends up getting used both for *mut T to *const T and *const [u8; 4] -> *const u8 -- something completely un-scary -- but also to quietly go from *const T to *mut Tcausing (or at least being a major contributor) a soundness issue in core.
Like how clippy can help use into instead of as, I hope we one day get more targeted methods for the less-scary cases, and then maybe the sketchier as casts here can move into something like a safe transmute API, where it should be more obvious that even if safe they might need to be handled with more care. (Similar to how such an API will eventually allow transmuting u32 to [u16; 2], but such a thing needs to be handled carefully to be portable and have any sane meaning.)
Which would make the whole round-trip transformation rather pointless.
The problem with using cfg!(target_endian = "big") or similar is that the solution only works on that class of architecture. So if you're designing only for x86 or x86_64, then cfg!(target_endian = "little") may make sense, but if the code is also supposed to work on big-endian architectures then the transmute really is uselss AFAICT.