Fast File Concatenation

Is there anything more performant than doing io:: copy if I need to cat together many large files?

I see fs::copy, but that overwrites the destination.

You could try copy_file_range (man page) which avoids some back-and-forth kernel calls, and fs::copy does use this when the kernel is new enough. There's an open issue to try using this in io::copy too, but it would have to be specialized since not all Read/Write types are file descriptors. You can control that in your in own case.

2 Likes

Nice! I'll look into that. I was surprised that nobody really seems to have a 'cat as a library' crate out there since this is the kind of simple thing that gets tricky to do correctly at high performance.

I ended up just making the buffer sizes of my reader and writer much larger based on this blog post: https://eklitzke.org/efficient-file-copying-on-linux

And that sped things up a lot, still using io::copy

let mut reader = BufReader::with_capacity(0x20000, in_fh);
let mut writer = BufWriter::with_capacity(0x20000, out_fh);
io::copy(&mut reader, &mut writer);
4 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.