GATs don't wholly solve the problem because &T and &mut T are fundamentally quite different, and the compiler has to use different reasoning with them. mut is more than just a modifier on top of &. That doesn't mean the idea is without merit. There are a number of earlier discussions on this topic; searching for abstract over mutability has some promising results. Also check IRLO (that is, internals.rust-lang.org); it's been floated over there as well.
It's unclear how exactly you suggest using raw pointers, but I suspect it would violate the rule that casting an &T to an &mut T is insta-UB, even if the &T was originally created from an &mut T, and even if the &mut T is not used to modify anything.
Yes, you are using addr_of! and that is ok in this case, but people often do this wrong, going through one of the reference types. Additionally, there's no addr_of! equivalent for things like a HashMap lookup.
Introducing mut generics wouldn't help in that case either, unless HashMap was expanded to take advantage of it...
But that seems kinda niche, and you might be able to get away with UnsafeCell in that case. (no transmute tho. do not transmute. transmute bad. love how UnsafeCell doesn't have niches so it isn't actually transparent. but uhh anyway we digress.)
Hence the problem: it's not that abstracting over mutability is impossible, it's just not as useful as one might hope, because while it may be mildly helpful for trivial functions, it's no help at all for complicated ones (which are the ones you most want the compiler's help with).
It might be nice if there was a :mut fragment specifier for mutability in declarative macros, which would basically match either mut or no mut (similar to :vis). Edit: That's just $(mut)?, and is kind of a half measure itself, so ehh.
but the devil is in the details: reborrow and borrow_mut don't return the same type modulo mutability, because while reborrow can be called on any kind of reference, borrow_mut can only be called on an owned value. In fact, the node handles defined in btree_map::node have not just two or even three, but five kinds of borrowing. The only reason these methods are as short and sweet as they are is because somebody's already done the work of hiding all the unsafe behind an internal API that abstracts over borrowing type in a way that isn't generalizable, but actually is just specific to BTreeMap. Of course, this is done with a substantial amount of unsafe code, which brings me to my next point:
unsafe is exactly the code where it's most important to distinguish clearly between &T and &mut T. It's already quite easy to break aliasing rules with unsafe and not know you're doing it; writing an algorithm that has to be valid for both is even harder (using addr_of!, etc). So I think if you're mixing in unsafe, a built-in method of abstracting over mutability is probably more of a footgun than anything else. Here's a previous discussion in which I felt @Yandros made this point well:
I'm open to finding out I'm wrong here -- I think it'd be great if there's a way to add parametric mutability in a way that makes complicated stuff like BTreeMap simpler! But that's much more complex than a syntax-based transform like @VorfeedCanal's, and even the type-based solution suggested by @Mokuz doesn't really begin to address the problem of implementing a data structure with unsafe -- if anything, it might even make it worse.
By the way, I didn't pick BTreeMap deliberately because I knew it would be this complicated. I just like to remind people it exists.