In the caller, the compiler knows whether or not to insert a reborrow before actually running the borrow checker. Here's a post I wrote a while ago on a related topic. Reborrowing can only happen at a coercion site where both the actual type A and the expected type E are of the form &'_ mut T, with (potentially) different lifetimes in the '_ slot. If E is generic, the compiler will simply choose E = A, and the value will be moved instead of reborrowed. This matches how other types of coercions work, and is why you don't have to specify types at every possible coercion site. The compiler will not choose E as some other type than A in order to allow a reborrow to happen.
In your earlier example you can make the test1 call compile by writing test3::<&mut _>(xref), which hints to the compiler to use a fresh lifetime for monomorphizing test3, so xref will be reborrowed instead of moved. (Of course you could also just explicitly reborrow: test3(&mut *xref).)
Could we invent a new rule that allows reborrowing everywhere? Maybe (I'm not convinced there isn't some scenario where you need to move). But that would be more complicated than the current situation, not less. Would the language be just as expressive and less complicated without the implicit reborrow rule? Yeah. But it would mean you have to use &mut * a lot and that would also be hard to explain to new users. The reborrowing rule makes sense in the vast majority of cases. Anyway, that would be a different language. (If you're inventing a new language based on Rust but without implicit reborrowing, consider also eliminating . for method call syntax; it makes things way simpler.)
tl;dr Yes, implicit reborrowing is a little quirky in how it interacts with generics, but it's useful in other ways.