System level cache via mmap

Hi guys,

I will need a cache system based on a Hashmap (ahash) that can be read from multiple processes on the system it is running on (Linux).
I want a login management for multiple apps on the system.

The basic idea, is a hashmap using a custom allocator, with proper locks, using a file as a mmap region to share the cache from multiple processes.

I already have a custom db that does this but the question is that I would like to reuse as much already existing pieces as possible.
I want to compare my custom solution with a custom hashmap.

Using a custom allocator should make it but I want the opinion of people with a better understanding of Rust than I do before doing anything.

The need:

  1. A shareable hashmap cache to multiple processes on the system
  2. Linux
  3. Use a file based mmap
  4. Enables me to write the allocation logic, locks..
  5. Reuse the ahash algorithm for the AES instruction

My current vision:

  1. I just finished writing a db that does exactly this, but I want a hashmap version to compare.
  2. I will write a custom allocator ( is it something stable enough or should I expect major hurdles
  3. Never wrote a custom allocator (for a std collection container) in Rust (but in other languages). I am a bit hesitant.
  4. If there is a slightly easier path / already done crate that does exactly this, I am all ears.
  5. performance will be reduced <-> synchronization: I am perfectly aware of the loss of performance, but shareability is paramount here. A fast hash could help a bit -> ahash.
  6. I could launch the apps in the same process and benefit from the shared memory here, but I may launch different apps at different time. So.. nope.
  7. I would like to avoid forking from Hashmap to rewrite anything. Too long. Reuse is the goal. I want to leverage the experience of experts.

I am asking for your vision, ideas, experience of what is expecting me with the borrow checker or anything.

Thanks guys

Why aren't you just using a file-backed database out of the box, eg. SQLite? That'll handle all the locking, memmapping, etc. for you.

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.