Certainly the second point you make is true. Regarding the first point, I'm not even sure the compiler will let you have an invalid reference, so I don't think that could even happen in a compiled program.
The best place I've found to learn about references and how they also affect access (also with regard to ownership) is in the Programming Rust by Blandy et al, that aren't mentioned in the online book but are super important.
For example, what I didn't know was that you can declare a value as immutable, but then later borrow a mutable reference to it, and modify it. <- EDIT: This is not true, I was mistaken.