I need some tips on reading and writing files. What I did was to compare IO speed between rust and perl. What I got were the following results:
Rust:
fn ge_read_files(file: &str)-> io::Result<()>{
let mut out = File::create("out")?;
let mut writer = BufWriter::new(&out);
for line in io::BufReader::new(File::open(file)?).lines() {
let mut line = line?;
line += "\n";
writer.write_all(line.as_bytes())?;
}
Ok(())
}
cargo build --release
time ./test input.txt // input is app 1.4GB
real 0m16,433s
user 0m10,801s
sys 0m2,487s
And the same with perl:
Perl:
time perl -lne 'print $_' input.txt > out
real 0m27,086s
user 0m21,824s
sys 0m2,431s
Rust did it 2x faster than Perl, but now I am wondering whether this is it or is there a better more faster way to do this in Rust? Is there any room for improvements ?
yes ... the example is not the cleverest i admit i am trying to see whether there is a faster way to go through an entire file (read it ) and to write to a file.... middle processing is not important now but here serves more as a placeholder that needs to be here and it can be anything (regex, append stuff, etc...). in my example i did just copy the two, but i could have just as easily created a loop to write stuff into a file withought first reading it. What I am interested is to see what would be the fastest way to read files (line-by-line) and write files (line-by-line).
I think the fastest way to copy files will be by using memory mapped files, see for example memmap crate. Though the main bottleneck here usually will be hardware, not software.