What's a recommendation for a database for a cache?
Would prefer it be entirely in safe Rust.
Interesting setup. Can you tell us a bit more about your use-case? A 50GB cache makes me wonder, do you intent to store the cache on-disk (i.e. because your database runs on a different machine and your network latency is too high so that retrieving the cached value from disk is measurably faster)? Or do you have a machine that has that much memory to spare for an in-memory cache? Also, why do you prefer a DB written in Rust over well-established key-value databases not written in Rust, like Redis, Memcached, etcd, Riak, etc.?
It's the content cache for a metaverse client. All the content is fetched from the servers, but has to be cached locally or it takes minutes for a user to log in and get a clear view of the world. Nothing is ever changed; keys are UUIDs and the associated contents never changes. It just gets discarded when stale.
It's definitely an on-disk cache, usually an SSD today.
The current implementation is a directory with tens of thousands of files in it, which means too much time is spent in the OS opening and closing files.
Redis is overkill. Memcached is client/server. Riak is distributed and in Erlang. I don't need anything that complicated. This is purely a local database. It's a relatively minor part of the system.
What I need is like the storage engine inside a caching server such as Ngnix or Varnish, minus the client/server stuff.
I was also looking for solutions for key value stores in Rust a while ago. For my own uses, I created mmtkvdb (which uses LMDB as a backend), but it's using memory mapped files and can exhibit UB if the storage is corrupted (which is why it also requires using unsafe when opening a database). Moreover, it hasn't been thoroughly reviewed.
Something that crashes on bad data is not helpful here.
I already have two crates on which I rely that are causing crashes. I'm not going to name names in this topic, but currently I spend more time on bugs in lower level crates than I do on my own code.
Yeah, hence my disclaimer. I think this also rules out a lot of other non-Rust DBs.
I'd be interested in a safe-Rust key value store as well.
Have you tried using several levels of directories so that no single directory has a lot of files? This might perform better.
There's always that approach. There's still a lot of file opening and closing involved, but the lookup times improve.
I was hoping there was something simple I could use. This is just key-value of blobs; the database never looks at the contents of the data.
Re-thinking about this, I don't think that most databases result in UB because of corrupted storage (but I'm not certain). In many cases, they would just abort, I guess (which is still better than UB). But maybe there's also databases around which provide even better error handling on a corrupted state and/or improper API usage.
However, I feel like when you have C APIs, it's rarely documented what happens when certain prerequisites are not met. So I do understand the wish for a pure (safe) Rust solution.
Would SQLite be an option? I know it's not specifically just a key-value store, but it's trivial to use as such. It's in-process, has a de facto Rust crate, it's highly concurrency-safe and hard to corrupt, and it's usually faster than the filesystem for small BLOBs, exactly because it avoids re-opening files for every lookup.
You can run Redis on machine with enough swap space. It would be slow as heck if you need in memory cache but it doesn't seems so.
Do you know how SQLite behaves when the on-disks storage is corrupted? Will its API return errors or will it abort the process? I see there is an SQLITE_CORRUPT error code, but I wonder if it's guaranteed that this code or other codes will be returned for all sort of corruption, and that there exists no state of the database, which will lead to an abort of the process or an endless loop, deadlock, etc. (I believe ideally that should be the case, but this seems to be difficult to judge about, I guess?)
It is documened to be guaranteed, or at least the authors intend to guarantee it. SQLite is one of the best-tested pieces of free software in the world right now, and it's tested (including fuzzing) to ensure that corrupt database files and user errors do not cause random crashes but reported deterministically as errors.
Naturally, there are kinds of corruption that it can't protect against. For example, if the DB file is directly overwritten in just the right place so that a value is changed but it is otherwise valid/looks "correct", then this is impossible to notice in the absence of some other explicit redundancy mechanism (e.g. value/row hashes).
Also an interesting link concerning how you can corrupt an sqlite file: How To Corrupt An SQLite Database File
So then this might be very much suitable for the purposes of the OP.
Which Rust wrapper would you recommend on that matter? I see the Rust Nursery lists rusqlite. Not sure how much up-to-date that site is, though.[1] Which would you recommend?
The last commit to the master branch of the Rust Nursery has been 2021. âŠī¸
Rusqlite is in fact the de-facto standard crate I was referring to. To my knowledge, it is the most popular and best-maintained SQLite wrapper in Rust. (I don't fully agree with all of its design choices, though.)
I like the whole sqlx framework, which also comes with a built-in sqlite3 driver. It is very comprehensible, async and does not require much overhead IMO.
Here's a minimal example of using SQLite as a key-value store. It has the following features:
Eq bound to ensure they are well-behaved.Borrow-based pattern for keys and values that std's map types apply. Thus, a Collection<String, Vec<u16>> can be created and accessed using &str and &[u16] as well, for example.chrono crate, and a None expiry date means that the given entry never expiresserde_json is hand-optimized and very fast, encoding can certainly be improved further by means of a binary, compact serialization format, such as bincode, MessagePack, BSON, or CBOR. (These are not used in the example because none of these crates seems to be available in the Playground.)It looks like there's some consensus for SQLite. It does have a good reputation for stability and a huge number of users.
This Stack Overflow article indicates it's possible to put blobs into SQLite without JSON encoding them.
Does that work in Rust? Applying JSON encoding to gigabytes of images would slow things down.
My implementation already stores keys and values as BLOBs, which should be clear from the included SQL. The serialization/deserialization layer is only there to allow arbitrary serializable types in the interface. You don't have to perform the serialization if all you ever have is raw bytes.