How fast can IO operations go?

#1

Hi,

I need some tips on reading and writing files. What I did was to compare IO speed between rust and perl. What I got were the following results:

Rust: 
         fn ge_read_files(file: &str)-> io::Result<()>{
                let mut out = File::create("out")?;
                let mut writer = BufWriter::new(&out);
                for line in io::BufReader::new(File::open(file)?).lines() {
                      let mut line = line?;
                      line += "\n";
                      writer.write_all(line.as_bytes())?;
                }
               Ok(())
         }

cargo build --release

time ./test  input.txt    // input is app 1.4GB

 real  0m16,433s
 user 0m10,801s
 sys   0m2,487s

And the same with perl:

Perl:
 time perl -lne 'print $_' input.txt > out

 real	0m27,086s
 user	0m21,824s
 sys	0m2,431s

Rust did it 2x faster than Perl, but now I am wondering whether this is it or is there a better more faster way to do this in Rust? Is there any room for improvements ?

thnx

#2

Seems like what you’re doing is actually just:

io::copy(
    &mut File::open(file)?,
    &mut File::create("out")?
)
1 Like
#3

yes … the example is not the cleverest i admit :slight_smile: i am trying to see whether there is a faster way to go through an entire file (read it ) and to write to a file… middle processing is not important now but here serves more as a placeholder that needs to be here and it can be anything (regex, append stuff, etc…). in my example i did just copy the two, but i could have just as easily created a loop to write stuff into a file withought first reading it. What I am interested is to see what would be the fastest way to read files (line-by-line) and write files (line-by-line).

thnx for the comment

#4

I think the fastest way to copy files will be by using memory mapped files, see for example memmap crate. Though the main bottleneck here usually will be hardware, not software.

#5

I think on linux that would be sendfile which bypasses userspace entirely. Not an expert in this domain though, might be wrong!

#6

Well, if we’re copying without changing, then hardlinks are going to be even faster :slight_smile:

Some filesystems (like APFS on macOS) have special APIs for hardlink-like copy-on-write file duplication.

#7

Here’s a crate that works with stable Rust: zio-sendfile.

2 Likes