There is a confusion here: for Vec<Point> there is no such thing as "changing where some entry points to". This Vec's elements are the Point objects, not pointers to Points. This has implications as you will see below.
You can get roughly what you want via a slice reference:
The second case needs to be allowed if the third and fourth are allowed because assignment is a particular case of a mutating operation. It doesn't assign a pointer to a new Point object located elsewhere -- it mutates the Point by replacing it.