Is there anything more performant than doing io:: copy if I need to cat together many large files?
I see fs::copy, but that overwrites the destination.
You could try copy_file_range (man page) which avoids some back-and-forth kernel calls, and fs::copy does use this when the kernel is new enough. There's an open issue to try using this in io::copy too, but it would have to be specialized since not all Read/Write types are file descriptors. You can control that in your in own case.
2 Likes
Nice! I'll look into that. I was surprised that nobody really seems to have a 'cat as a library' crate out there since this is the kind of simple thing that gets tricky to do correctly at high performance.
I ended up just making the buffer sizes of my reader and writer much larger based on this blog post: Efficient File Copying On Linux
And that sped things up a lot, still using io::copy
let mut reader = BufReader::with_capacity(0x20000, in_fh);
let mut writer = BufWriter::with_capacity(0x20000, out_fh);
io::copy(&mut reader, &mut writer);
4 Likes
This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.