As one data point, this is exactly what WebAssembly does.
Once you peel away the layers of abstraction, linear memory is just a Vec<u8> and pointers are just indices into this vector. There are even instructions like core::arch::wasm32::memory_grow() that let you tell the host application to increase this Vec<u8>'s size when you want more memory (i.e. Vec::reserve()), which wouldn't be possible if pointers inside WebAssembly code were "real" pointers.
So @bread-dreams, I guess if it's good enough for WebAssembly it should be good enough for you, too
Even "real" pointers work similarly under most operating systems: Each process has a private address space and the virtual memory system is responsible for translating these private addresses to physical memory locations, which can be moved around at any time by the OS. The brk syscall extends this private address space, the same as WASM's memory_grow.
