Actually, I should add that you can still do it your way, minimizing the scope of the unsafe by moving a reference to the data within the unsafe block:
&(unsafe { &*self.pointer }).0
Here, you are moving a borrowed reference and .0 auto-derefs that reference.
See the second example in this playground link.
Like, I understand why this happens after being given an explanation, but... It feels as if there could be a way to make this simpler (maybe just an improvement of the error message for this particular case - no change of language semantics involved)