I would think that many optimizations will depend on knowing that the input is valid first, even with respect to borrowing, or else that transformation might be invalid itself.
No. For a trivial example, optimized MIR can remove dead code, and you still want checking on that despite it being dead -- especially if it's only dead after inlining, or something. (See a bunch of the bugs that happened with the unsafe checks moving to MIR, where for a while under a particular nightly feature flag you could call unsafe methods in safe code without an unsafe block so long as they were dead, IIRC.)