Fs::walk_dir causes too many open files


#1

Hi,

I’m new to Rust so my approach here might be wrong but here goes. I’m trying to use walk_dir to traverse a fairly large directory structure and I run into (on Mac) that I get “Too many open files” back after a while. My code looks like this

match fs::walk_dir(target_path) {
    Err(why) => { 
        println!("Unable to walk_dir {:?}", why.kind());
        return;
    }

    Ok(paths) => for path in paths {
        match path {
            Err(err) => { 
                println!("Unable to get path: {}", err);
                return;
            },
            Ok(fname) => println!("name {}", fname.path().to_str().unwrap()),
        }
    },
}

The output goes like this

name file1
name file2
...
Unable to get path: Too many open files (os error 24)

It seems like file handels are being leaked or just that too many are being tried to be open at the same time. I wonder if there is some good way to solve this or do I need to do some manual traversing in order for it to work?

Cheers!


#2

It looks like it keeps a stack of all sibling directories found and remaining to visit, opened. Would be better to open them when visiting.


#3

Yeah I actually (of course after I posted here) found this https://github.com/rust-lang/rust/issues/23715 so it seems to be a known issue already.

I will work around the issue for now. Thanks for the reply!