I'm currently building a system which at a point needs to list every single file on a given partition and get its name, size in bytes, creation and modification date.
I've been able to build a list of all files in a partition very quickly thanks to WalkDir with very good performances (only a few seconds for ~ 100k files).
But the problem I have is the metadata fetching. Calling the
Path::metadata function seems to be quite costly, even though it's fast on a single file, it takes more than 5 minutes to complete for 100k with parallel iteration (rayon) on 8 physical cores.
I know it's possible to get results a lot faster than that, even a simple
ls **/* is faster (about 2 minutes, and it also fetches the permissions and displays them in the terminal), and specialized tools like WizTree can do so in a few seconds only.
So I'm wondering, what is the fastest way to fetch basic metadata on every single item in a given filesystem? I guess the best would be to directly read at the filesystem's files table like WizTree does, but I didn't find any crate able to do that.
I'm currently stucked as the final tool needs to run on filesystems with nearly a million files, which take way too long.
Thanks in advance for your help!