I think libpmem is a good variant for disk mapper for VM memory, as it has some perspectives in Intel Optane usage rising in future, and SSD we already have now.
The first problem I see is a lack of (index-)based pointers in Rust.
Just not saying about the custom allocator required and optimized for SSD wearing.
What I mean:
structures should be able to be marked as persistent, so
memory for them must be allocated in a special RAM area mmaped to a disk file
and the most complex thing: access to these structures should be done by based references, which has two parts:
index in the array of pointers to mmap areas (to let objects move between multiple storages)
platform- and memory-independent address relative to the start of mmaped (VMEM) area
The problem is: every sequential run of the engine has a floating address of VMEM, as OS does not guarantee mmap fixation in process address space between process shutdowns/restarts, and run of virtual memory image on different OSes.
Generally, it does not matter a big for classical bytecode interpreters, as addressing complexity is hidden in relatively small code, and programmers almost never have any interaction with raw pointers in memory.
But I want to use Rust itself intensively, and relative addressing will increase syntax complexity for all code that interacts with data relocated in VMEM.
It doesn't sound like any of these problems are specific to Rust, so you should be able to look at a C or C++ implementation and reuse most of the same tricks/techniques.
There are plenty of Rust libraries for working with mapped memory. I'm guessing you'll end up calling the underlying C functions (e.g. libc::mmap) instead of a high level wrapper like memmap so you can tailor the code to your particular use case.
This is just a case of writing your own smart pointer which encapsulates all the logic around relative addresses so the "syntax complexity" isn't really an issue. If you implement the std::ops::Deref trait your smart pointer can be passed around and treated largely identically to a normal reference.
It's not too complex, although you may have to be careful to ensure you don't have dangling pointers.
One thing you'll need to be careful about is that safe Rust's memory model generally forbids use of memory-mapped data, if that data could change from outside of the process.
&[u8] is totally immutable, no exceptions. &mut [u8] can only be changed by the program itself.
&[UnsafeCell<u8>] is permissible, or you could use ptr::read_volatile() on the mmapped data.