How fast can IO operations go?

Hi,

I need some tips on reading and writing files. What I did was to compare IO speed between rust and perl. What I got were the following results:

Rust: 
         fn ge_read_files(file: &str)-> io::Result<()>{
                let mut out = File::create("out")?;
                let mut writer = BufWriter::new(&out);
                for line in io::BufReader::new(File::open(file)?).lines() {
                      let mut line = line?;
                      line += "\n";
                      writer.write_all(line.as_bytes())?;
                }
               Ok(())
         }

cargo build --release

time ./test  input.txt    // input is app 1.4GB

 real  0m16,433s
 user 0m10,801s
 sys   0m2,487s

And the same with perl:

Perl:
 time perl -lne 'print $_' input.txt > out

 real	0m27,086s
 user	0m21,824s
 sys	0m2,431s

Rust did it 2x faster than Perl, but now I am wondering whether this is it or is there a better more faster way to do this in Rust? Is there any room for improvements ?

thnx

Seems like what you're doing is actually just:

io::copy(
    &mut File::open(file)?,
    &mut File::create("out")?
)
1 Like

yes ... the example is not the cleverest i admit :slight_smile: i am trying to see whether there is a faster way to go through an entire file (read it ) and to write to a file.... middle processing is not important now but here serves more as a placeholder that needs to be here and it can be anything (regex, append stuff, etc...). in my example i did just copy the two, but i could have just as easily created a loop to write stuff into a file withought first reading it. What I am interested is to see what would be the fastest way to read files (line-by-line) and write files (line-by-line).

thnx for the comment

I think the fastest way to copy files will be by using memory mapped files, see for example memmap crate. Though the main bottleneck here usually will be hardware, not software.

I think on linux that would be sendfile which bypasses userspace entirely. Not an expert in this domain though, might be wrong!

Well, if we're copying without changing, then hardlinks are going to be even faster :slight_smile:

Some filesystems (like APFS on macOS) have special APIs for hardlink-like copy-on-write file duplication.

Here's a crate that works with stable Rust: zio-sendfile.

2 Likes