Fs::metadata in Windows unnecessarily slow

help
1

I spent the last couple weeks porting a C++ build manager app to Rust. When it was done, I had a significant performance degradation that I spent several days tracking down. It came to this: fs::metadata() in a Windows app is opening the file to get a handle, which is unnecessary. I use it to get a bunch of files' last modified times. In C++, there's a std::filesystem::last_write_time(), which runs 5x faster than the Rust equivalent fs::metadata().unwrap().modtime().unwrap().

If you compare the source code, you can see that the C++ version is calling a Windows API function GetFileAttributesExW(), which does not require a file handle and therefore an open file. I think everything you need to fill out the std::fs::Metadata is available from the WIN32_FILE_ATTRIBUTE_DATA object the Windows API operates on. If so, this is a performance degradation for no good reason.

I worked around the problem, but it required some ugly, unsafe things.

2

For reference, it did use GetFileAttributesExW until:

1 Like
3

Note that fs::metadata is required to follow symlinks (which on Windows we interpret as other type of links too) so it would need a fallback. However, a fast path does make sense.

I think everything you need to fill out the std::fs::Metadata is available from the WIN32_FILE_ATTRIBUTE_DATA object the Windows API operates

It doesn't get the reparse tag (used to disambiguate links from other types of reparse point). Though this only matters if it's a reparse point. It would mean it's doesn't get some information which is currently nightly-only. E.g. MetadataExt in std::os::windows::fs - Rust

4

Plural? Other than the call to GetFileAttributesExW, what else was marked unsafe?

5

I didn't find a good way to get the FILETIME value into the opaque SystemTime, so I transmute()d it

1 Like
6

Ideally we'd use NtQueryInformationByName which is both faster and provides a lot of information. However, it's a very new function (only ~7 years old). It'd still need a fallback for links though.

7

The correct way is to add a Duration to SystemTime::UNIX_EPOCH.

8

Do you have a recipe for that? I've tried a couple approaches and am not succeeding at getting the two u32 high/low values into a Duration that yields an equivalent value to the exiting SystemTime calculation. In the source, SystemTime is defined as

pub struct SystemTime {t: c::FILETIME, };

So I feel like the transmute value I'm getting now is yielding the correct answer.

Neither of these has worked:

// attempt 1
let duration = std::time::Duration::new(attrs.ftLastWriteTime.dwHighDateTime as u64, attrs.ftLastWriteTime.dwLowDateTime);
// attempt 2
let as_u64 = ((attrs.ftLastWriteTime.dwHighDateTime as u64) << 32) | (attrs.ftLastWriteTime.dwLowDateTime as u64);
let duration = Duration::from_secs(as_u64);

let systime = SystemTime::UNIX_EPOCH.checked_add(duration);
assert_eq!(systime, Some(std::mem::transmute(attrs.ftLastWriteTime));
9

They're in units of 100ns, and they start from January 1, 1601.

So something like:

let as_u64 = ((attrs.ftLastWriteTime.dwHighDateTime as u64) << 32)
    | (attrs.ftLastWriteTime.dwLowDateTime as u64);
let duration = Duration::from_nanos(as_u64 * 100);
let unix_epoch_minus_windows_epoch = Duration::from_secs(11_644_473_600); // from std: https://github.com/rust-lang/rust/blob/99d0186b1d0547eae913eff04be272c9d348b9b8/library/std/src/sys/pal/windows/time.rs#L27

let systime = SystemTime::UNIX_EPOCH + duration - unix_epoch_minus_windows_epoch;

Honestly this is kind of annoying and I'm not sure it optimizes out. Maybe just include it as a test to make sure the transmute is valid.

1 Like
10

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.