Consider what happens if you moved out of a mutable reference, then panic. Then the panic unwinding is caught up the call stack in the scope of the original owner of the object you are borrowing. Through the unwinding the mutable reference is dropped and so the original owner could access the object (by this i mean the compiler would let you)... But wait there is nothing there anymore (technically that memory stores something but it may not be valid anymore). Accessing the moved out of object would be UB and unsound. Tracking moved-out-of-ness would require a bunch of runtime overhead (basically what c++ does) so the compiler just doesn't let you move out of a &mut
Things like swap/take/replace know there won't be a panic between moving out and moving in a new object, but they still need to use unsafe as this is not provable to the compiler. But it is a safe abstraction you can build on.
The borrow checker doesn't treat Drop in any special way. It's just any other &mut self method. You couldn't pass ownership of self to drop because then at the end of drop, the destructor of self would run ... which is a bit problematic when you're in the destructor.