Fast iteration on files' metadata

Hi there!

I'm currently building a system which at a point needs to list every single file on a given partition and get its name, size in bytes, creation and modification date.

I've been able to build a list of all files in a partition very quickly thanks to WalkDir with very good performances (only a few seconds for ~ 100k files).

But the problem I have is the metadata fetching. Calling the Path::metadata function seems to be quite costly, even though it's fast on a single file, it takes more than 5 minutes to complete for 100k with parallel iteration (rayon) on 8 physical cores.

I know it's possible to get results a lot faster than that, even a simple ls **/* is faster (about 2 minutes, and it also fetches the permissions and displays them in the terminal), and specialized tools like WizTree can do so in a few seconds only.

So I'm wondering, what is the fastest way to fetch basic metadata on every single item in a given filesystem? I guess the best would be to directly read at the filesystem's files table like WizTree does, but I didn't find any crate able to do that.

I'm currently stucked as the final tool needs to run on filesystems with nearly a million files, which take way too long.

Thanks in advance for your help!

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.