The single most important Rust performance tip is simple but easy to overlook: make sure you are using a release build rather than a debug build when you want high performance. This is most often done by specifying the --release flag to Cargo.
A release build typically runs much faster than a debug build. 10-100x speedups over debug builds are common!
for file in _files.into_iter() {
let file_path= file.unwrap().path();
- if file_path.is_dir() && !file_path.is_symlink(){
+ let meta = std::fs::symlink_metadata(&file_path).unwrap();
+ if meta.is_dir() {
dirs.push(file_path);
}else{
files.push(file_path);
}
}
(You're calling stat on every entry (resolving symlinks) and then lstat on every directory; the above calls lstat on every entry.)
btw, do you execute sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' before the both program execute?
the program runs first should be slower since it does not gain benefit from the cache.
I don't know about the Node version, but as an iterator, WalkDir probably isn't collecting things into Vecs which you need to allocate, populate, and then afterwards iterate over (and probably deallocate). If you made your version recursive, you could remove the dirsVec, but you'd still be creating then separately iterating the files one.
This matters on my machine. I just copy and run the code given by OP.
$ cargo r -q -r # first time
281483
==13159ms
$ cargo r -q -r # second time
281483
==3959ms
$ cargo r -q -r
281483
==1443ms
$ cargo r -q -r
281483
==1109ms
$ cargo r -q -r
281483
==1137ms
$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ cargo r -q -r # first time
281483
==18544ms
$ cargo r -q -r # second time
281483
==1927ms
$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ cargo r -q -r
281483
==18747ms
$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ cargo r -q -r
281483
==18557ms
struct GetFiles {
stack: Vec<PathBuf>,
current: ReadDir,
}
impl Iterator for GetFiles {
type Item = PathBuf;
fn next(&mut self) -> Option<Self::Item> {
loop {
if let Some(file) = self.current.next().and_then(|f| f.ok()) {
let file = file.path();
if fs::symlink_metadata(&file).map(|m| m.is_dir()).unwrap_or(false) {
self.stack.push(file);
} else {
return Some(file);
}
} else {
let dir = self.stack.pop()?;
self.current = fs::read_dir(dir).ok()?;
}
}
}
}
impl GetFiles {
fn new(path: &Path) -> Self {
let stack = <_>::default();
let current = fs::read_dir(path).unwrap().into_iter();
Self { stack, current, }
}
}
fn main() {
let sys_time = SystemTime::now();
let count = GetFiles::new("../../".as_ref()).count();
println!("{count}");
println!("=={:?}ms", SystemTime::now().duration_since(sys_time).unwrap().as_millis());
}
Though it didn't make a big difference for me. The caches @Neutron3529 mentioned make a big difference though, so be sure you're timing what you think you are.
thank you ,I tried running your code, same performance as 16s. I'm reading the source code of walkdir and I feel it's too complicated to write high performance 'FileSystem' code in rust
The following change (querying file type via DirEntry) resulted in significant speedup for me; at a guess the DirEntry has the metadata already and a syscall is avoided.
impl Iterator for GetFiles {
type Item = PathBuf; // DirEntry;
fn next(&mut self) -> Option<Self::Item> {
loop {
if let Some(entry) = self.current.next().and_then(|f| f.ok()) {
if entry.file_type().ok()?.is_dir() {
self.stack.push(entry.path());
} else {
return Some(entry.path());
}
} else {
let dir = self.stack.pop()?;
self.current = fs::read_dir(dir).ok()?;
}
}
}
}
I don't know how you were comparing with walkdir, but it returns DirEntrys and thus doesn't even create a PathBuf for everything (which inspired the above change -- I went back to returning PathBuf because it didn't make a big difference for me).
