I need a large number (potentially millions) of pointers to the heap. It would be useful if these could be ensured to use only 32 bits, even on x86_64 architecture. Is there any way to make the pointers 32 bit, or if not, to at least ensure the numerical value of the pointer has the most significant 32 bits as zero. I could always have a 64 bit pointer to a large section of memory, and use 32 bit indices, but I worry about the runtime performance cost when accessing the objects.
I don't know of any cross-platform way.
I think your best bet is indexing. This case is well known and optimized in LLVM and CPUs.
I thought of indexing but aside from pointer problems, I don't want to pass the base pointer (or Vec or whatever) around everywhere. Performance is really crucial, I tried good old mmap, which seems to work.
let big_heap = libc::mmap((1u64 << 16) as *mut libc::c_void, MAX_HEAP_SIZE, libc::PROT_READ | libc::PROT_WRITE, libc::MAP_PRIVATE | libc::MAP_FIXED | libc::MAP_ANONYMOUS, -1, 0);
Then all allocations have a low address, and I can safely cast the u32's to 64 bit pointers. It seems to work so far.
Not sure if this helps you in any way but the problem reminded me of Pointer Compression in V8 · V8
If you can ensure a new enough kernel (Linux 4.17), then
MAP_FIXED_NOREPLACE would be safer to avoid clobbering existing maps.
For x86-64 Linux only, there's also the
MAP_32BIT flag which only uses the first 2GB of memory space.
libc::MAP_32BIT is just what I was after.