I am asking about wasm, but not in the browser. Hacks (not officially supported by wasm standard or wasi standard) that work with only one of wasmer / wasmtime is fine.
I am writing a small DSL / VM in Rust that compiles to / runs on wasm -- and it is great so far. The only concerns I see are: (1) 4GB limit and (2) no way to do mmap (the fact wasm 'memory' is a 'linear memory' of the host seems to complicated this).
Question: is there anyway (ugly hacks fine) to break the 4GB limit and/or get mmap? I primarily use wasmer, but wasmtime solutions welcome too.
Wasm is a 32 bit architecture and this means that 4GB is not a soft limit, it's the whole addressable space. There's a proposal for a 64 bit version of wasm that would solve your problem, however it hasn't been finalized yet. Rustc has a wasm64-unknown-unknown target, but it's tier 3 and likely still unstable. Wasmtime has an open issue for wasm64 support, but it hasn't been implemented yet.
If you only need to read/write data in wasm you could implement and export functions like read_memory(offset, size) and write_memory(offset, data) in your VM to do your own paging. Even though data and size could be only 4 (or 2) GB, offset could be 64 bit. Alternatively you could expose chunks of predefined size and use an index instead of offset. Or have functions to access some in-memory database which could be more convenient than raw memory access.
I sincerely advise you to make like a 32-bit developer in 2008 and use file streaming algorithms, with the buffers managed by the runtime environment, for data over 2GB.
If you're using wasm for the sandboxing, the fact that we can reliably box its entire memory space is responsible for a great deal of its "safety at speed".
People have wrote the database on 32bit architecture without mmap the entire file. Check the std::io::Seek which allows to seek the file cursor using 64bit offset regardless of the ptr width.
Q: Is the size of the database restricted to the amount of physical memory in the system?
A: No. The maximum database size is constrained only by the amount of disk space available and by the size of the address space on the machine. For 32-bit implementations it’s restricted to approximately 2^31 bytes (2 GB)...
@Hyeonu : You have been extraordinarily helpful to me in the past -- but I don't understand where you are going with this line of thought.
For example, consider a machine with 16 TB of storage but only 64 GB of RAM.
Using 64bits, we can easily manage a DB of 10TB by memor mapping the 10 TB db-file, then having the OS cache layer take care of deciding what to bring into physical memory / deciding what to evict.
The reason I asked for wasm64 is that in wasm32, we have 32bit ptrs, so we can't memory map a file larger than 4GB, which means we can not (under the LMDB approach) have a > 4GB DB.
===
I consider this a pretty severe limitation, impractical for general use.
I don't know how familiar you are with LMDB. The general idea is that we view chop up the disk into 4KB nodes, then build a giant append-only b-tree. "modifications" are done by building a new version of the b-tree (with sharing) and swapping the root ptrs.
So in practice, we memory map the file backing the entire DB (which may be terra bytes), then we let the OS caching layer figure out what pages to bring into physical memory / what to evict.
Thus, the size of the file backing our database can not be bigger than our ptr size.
With wasm32, this limits us to 4GB -- which is very limiting.
What I wanted to say is that it's the inherent limitation of the database architecture you've chosen. The symas LMDB itself has 2GiB file size limit on 32bit arch which obviously is not sufficient for the large scale database. The reason it's 2GiB not 4GiB is that the x86 reserves the half of the address space for the kernel usage which is not applied to the wasm32. But I agree 4GiB is still too small.
I'm not saying you need to drop the LMDB arch. They're multiple requirements you've mentioned which is not possible as a whole. What you need is to drop at least one of them. You can
Drop the LMDB arch and seeking the file cursor instead.
Drop the WASM target and use those native 64bit archs instead.
Drop the size requirement. 2GiB would be descent enough for practical client side database.
Drop the "now" requirement and wait for the WASM64 format being stabilized. It may takes months, or years, but likely not decades.
The reasonable programmer adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable programmer.
-- George Bernard Shaw
I'm hoping somewhere out there is an unreasonable programmer who found the 4GB wasm32 limit unacceptable and came up with hacks to break the 4GB limit (thus the post title).
For wasm32 the load and store instructions take 32bit pointers and as such no more than 4GB of memory is addressable. Wasm64 is pretty much wasm32 except that load and store (and some other) instructions take 64bit pointers.
Any ideas on why adoption of wasm64 has been so slow? From a very naive perspective of only having played with cranelift/llvm slightly, this seems like a very straight forward addition of a few match statements for the 64 bit addresses
Where is the implementation complexity (that clearly exists but I am failing to see) at ?
If you map the VM's memory in such a way that the sandboxed code "physically"¹ can only express (offset) addresses of locations it should actually have access to², then you can both compile memory access without adding bounds checks and call into and out of Wasm without a context switch of any sort. This vastly improves performance in a platform-independent way.
I'm pretty sure it's possible to create an alternative implementation in a platform-dependent way using virtualisation features, but I haven't read too deeply into it.
My understanding is that it would take more effort to implement it this way. I'd like to hear if there are other limitations to that approach that I'm not aware of.
¹ The other way to take advantage of that would be to logically prevent programs from arbitrary reads (outside of anything that's clearly an array access, which would still need a bounds check) ahead of time, but as Wasm is mostly a "flat memory" runtime, doing this to a significant degree is very complicated and essentially limited to "known cases".³
² plus locations in unmapped pages, for which you can trap access in the runtime with little effort at zero cost
³ A truly general solution would solve the halting problem. Even with reasonably constrained programs, the search/evaluation space is extremely large and escalates exponentially.
