The LLVM backend of the rustc compiler places significant semantic constraints on the intermediate-representation (LLIR) code that the rustc compiler front-end produces. unsafe does not remove LLVM's input constraints; it merely instructs the rustc frontend that YOU are assuming some of the responsibility of meeting those constraints.
The term "Undefined Behavior" (UB) refers to the situation where you have violated LLVM's input constraints, giving it permission to do whatever it chooses with your code in its quest for hyper-optimization. If your use of unsafe leads to UB, you have given LLVM permission to miscompile your program. There's an old phrase for this: "Garbage in, garbage out".
Edit: Corrected "Unspecified" to "Undefined". Thanks, @lxrec