Std::fs::write / sync_all

Two separate comments in X86_64 linux, crash safe log has mentioned the question as off topic, so I am rewriting this question in purely Rust terms:

let a: &[u8] = ... ;
let b: &[u8] = ...;
let mut f: File = File::create("foo.txt")?;
power plug pulled

Now, let s be the contents of foo.txt. Is it guaranteed that:

  • len(a) <= len(s) <= len(a) + len(b)
  • first len(a) bytes of s is a

If no, how do we even built simple things (like the Logger) in Rust.

If yes, what guarantees the above invariant? Is it guarantees by the SSD, by the kernel, or by the rust std::fs::File impl ?

Some preliminary searches show that this looks like a bit of a rabbit hole.

The documentation of sync_all does not guarantee the flushing of hardware caches, probably because this would be too difficult to guarantee due to differences based on OS and user configuration.

Looking at Linux specifically, fsync's documentation says

This includes writing through or flushing a disk cache if present.

Which suggests that such functionality is part of fsync. (At least when write barriers (a kernel feature/mount option) are enabled)

However, there comes some additional questions. I've found some discussion on the state of things in 2012, where the answerer recommends using a drive with a non-volatile write cache: filesystems - Safety of write cache on SATA drives with barriers - Server Fault . These caches will flush their pending writes when power is restored to the drive.

I haven't found clear wording anywhere on whether this feature of the hardware replaces the need for kernel write barriers, or if it supplements them. Though there are some troubling comments on journaled filesystems on this page which lead me to believe they should be kept enabled: Chapter 22. Write Barriers Red Hat Enterprise Linux 6 | Red Hat Customer Portal

There is also discussion that fsync is very slow and should be avoided, but a lot of this discussion is old. I haven't found a modern perspective.

1 Like

It's not guaranteed by Rust. It can't be; it depends on things outside of Rust's abstraction zone / within the foundation Rust is built on: libc, the kernel, the FS, the hardware...

The Rust-level guarantee here is something like "Your buffers were flushed and fsync was called."

Maybe you want to use custom_flags to open your files with O_SYNC.

If the failure rate is too high, you start looking at the other layers, the guarantees and abstraction boundaries they have, calling their more-specific APIs if they exist, and putting restrictions in place (supported library versions, file systems, hardware)...

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.