If reading and creating a reference from a pointer to uninitialized memory is UB, then how does the compiler even understand that this memory is uninitialized?
In my program, I store addresses as use in another place, then make refs from them, then logically I will definitely encounter UB even if I previously initialized memory at this address.
In the general case, the compiler is allowed to blindly assume that memory accessed through a pointer is initialized. It doesn't - and can't - track which pointers point to initialized memory and which don't. It is your responsibility when working with bare pointers to ensure that only pointers into initialized memory are read (or converted into refs).
If your program were able to check before each pointer access, or if the compiler were able to reason about which pointers point to initialized vs. uninitialized memory, then there wouldn't be any need to leave the behaviour of accessing uninitialized memory through a pointer undefined.
In the limited case of using references, the rules of the language prohibit (safe) creation of a reference to anything other than initialized memory, so if you have a ref, the compiler's assumption must be true (or you have an unsafe block with a bug in it somewhere).
You will want to run your program under miri regularly when testing it, to catch mistakes around your own tracking of which pointers do or do not point to initialized memory.
In the general case, the compiler is allowed to blindly assume that memory accessed through a pointer is initialized.
And this applies to all UB. It is the reflection of some general assumption that a thing doesn't happen, because that assumption is necessary for some property, such as optimizations.