Truth be told, that's pretty much how I think of it already. As discussed a bit in Learning Rust with Entirely Too Many Linked Lists - #19 by vitalyd, it's already the case that you can have multiple aliases "resident in memory" - they just cannot be "active" (i.e. read/written) at the same time.
But the above gets into the whole business of "can a &mut be considered a token of having a unique access", even if you don't read/write through it. As you say: