Durably updating a file/block in 4kb blocks in pure rust

This is the driving force of a number of my recent questions, and perhaps this, instead of the low level questions, is the right question to ask.

  1. We are on x86_64 Linux.

  2. We want to write to some durable storage. This could be a file in the filesystem or a raw block device, like /dev/sda# -- you choice, whatever works best.

  3. A 'block' here refers to 4kb block of memory that starts are a 4kb alignment, i.e. [4096 * N, 4096 * (N+1)) for some N.

  4. We want to be able to efficiently read block N, write block N.

  5. We need this to be fast in the common case, but also be able to do crash-recovery due to crashes (OS crash, power plug pulled, etc ...)

Are there Rust crates that focus on solving this problem? (and nothing else, i.e. not 'pull it out from DB XYZ').

1 Like

My instinct says, "probably not", but this problem is language-agnostic enough that you can just look up a C implementation and write a simple wrapper using the nix crate.

Do you have one in mind / is there a name for this? If it's < 2k LOC, I might just rewrite it in Rust. This isn't exactly a "write ahead log"; it's almost like a bitcask/kv-store where all values are 4kb pages.

Not particularly.

It's not code and you've probably already found it in your search for a solution, but this Stack Overflow answer has some high level solutions and links to plenty of other resources.

1 Like

I think this is the first time I have read of "fsync dance".

Is there a book you recommend? (I feel like there is a gap in my knowledge), or is this stuff that only pops up with a postgres dev shows up and tells us the right way to do things ?

I don't normally learn from books, instead preferring to dig through things like API docs, GitHub issues, and source code.

I'm probably not the expert you are looking for anyway because my main exposure to the intricacies of filesystems is via random blog posts and articles... I know just enough to be dangerous, but not quite enough to be useful.

There was this really good article on LWN (used to be called "Linux Weekly News", I believe) about making filesystem operations reliable, but it's been a couple years since I read that and can't remember the title. You might want to skim through the articles tagged under "filesystems" to see if anything interesting pops up.