Fmmap: A flexible and convenient high-level wrapper mmap for zero-copy file I/O

Hi, fmmap is ready for use.

repo: https://github.com/al8n/fmmap

crate: https://crates.io/crates/fmmap

document: https://docs.rs/fmmap

fmmap aims to provide fast file I/O base on the memory map.

The design of this crate is inspired by Dgraph's mmap file implementation in Stretto.

All the file-backed memory map has the potential for Undefined Behavior (UB) if the underlying file is subsequently modified (e.g. the file is deleted by another process), in or out of the process, this crate tries to avoid this situation by providing file lock APIs.

This crate supports std and popular async runtime(tokio, async-std, smol), and thanks to macro in Rust, it is super easy to support any new async runtime. For details, please see the implementation for tokio, async-std, smol of the source code.

Features

  • dozens of file I/O util functions
  • file-backed memory maps
  • synchronous and asynchronous flushing
  • copy-on-write memory maps
  • read-only memory maps
  • stack support ( MAP_STACK on unix)
  • executable memory maps
  • file locks.
  • tokio support
  • smol support
  • async-std support

The async support isn't really async. Using your async types to read a memory mapped file is not going to yield to the runtime and will block the thread just like if you read the file with blocking IO.

1 Like

Can you please mark the open functions as unsafe? File locking is far from reliable. By the way are you using advisory or mandatory locks? I can't find the actual locking implementation through several layers of indirection.

1 Like

Does it make sure to catch any exceptions thrown from functions called by the code running in native client then? And then handroll a longjmp implementation to jump over this code. (the real longjmp on Windows uses SEH, so the hand rolled version has to save and restore registers like on Unix)

The file lock is use fs4 (original fs2). When you use the open function, it will not lock the file, whether locks the file depends on the user, there are 5 APIs to help lock/unlock the file.

For async types, if you want to use unblocking read or write, you should call reader, ranger_reader, writer, range_writer to get an async reader/writer. The basic read/write util functions in AsyncMmapFileExt/AsyncMmapFileMutExt is not unblocking.

I am not quite sure about this, but I think you may find the answer in the low-level dependencies, the low-level is based on memmap2 and fs2.

What are those methods? As far as I know, it's impossible to implement a proper async file object based on mmap, and any way of implementing it would be blocking.

1 Like

Thanks. Fs2 seems to use flock on unix which are advisory locks. Not only that but the user needs to explicitly opt in to using them. If you don't opt in trying to open the "locked" file succeeds.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.