As one data point, this is exactly what WebAssembly does.
Once you peel away the layers of abstraction, linear memory is just a
Vec<u8> and pointers are just indices into this vector. There are even instructions like
core::arch::wasm32::memory_grow() that let you tell the host application to increase this
Vec<u8>'s size when you want more memory (i.e.
Vec::reserve()), which wouldn't be possible if pointers inside WebAssembly code were "real" pointers.
So @bread-dreams, I guess if it's good enough for WebAssembly it should be good enough for you, too