Is there a crate which implements nested readers, aka SectionReader?

I need to store and later hand out a std::io::Read that reads from only a section of a file. There's a container format that I'm parsing, so I know the offset and length of the contained data. I think the most obvious solution is to simply store the current stream, the offset, and the length in a struct and implement std::io::Read for it, but that seems like something someone might already have done, and I want to skip shaving that yak if I can.

The Go standard library calls this a SectionReader, so that was the first thing I searched for, with no luck. Searching just for "reader" on cratesio finds 447 results, and I'm only about a quarter of the way through the list. I've found buffered_offset_reader, which is almost what I want (https://crates.io/crates/buffered_offset_reader). It's too low-level, but I could use it to implement what I want. The buffered_reader crate (https://crates.io/crates/buffered-reader) has nested readers, but they are only temporary. That is, you have to read from the nested reader and then discard it; the nested reader simply reads from the current position in the file rather than seeking to an offset first.

I'm not aware of something like this.

I think you're looking for std::io::Read::take.

Not quite. If I call f.take(5) and then stash that somewhere for an hour, when I do need to use it it will let me read 5 bytes from whatever position the file descriptor happens to be pointing to at the time. It doesn't save the start point, only the length.

I see. But it gets you half of the way there. You could open a new file handle for each section.

Otherwise, implementing the equivalent of Go's section reader seems like it should be very easy. std already exposes the necessary platform functions for Unix and Windows. From there, it looks like a little glue code is all that's needed to wire everything up.

I can't count on being allowed to have that many open file handles, but otherwise that would be a simple solution.

The alternative is seeking back and forth.

I've implemented something that could help.
The FragmentedReader takes T: Read + Seek, a list of offsets + length (Fragment) and implements std::io::Read and std::io::Seek.
https://github.com/nickelc/hpk/blob/master/src/hpk/read.rs#L23

No, it's not. As I shared in my previous comment, you can use the platform specific read_at and seek_read APIs to implement this with no explicit seeking at all. read_at on Unix uses pread and and seek_read on Windows uses ReadFile.

This is presumably how Go does it and it can be done without ever calling seek.

Yes, that's why I included another suggestion.

That's cool. Your stream is made of many fragments that are read in order? In my case the data is contiguous, so I would have only a single fragment. Aside from that we implemented the read logic in exactly the same way, which is cool.

I wasn't ignoring it, but it's the same as what I mentioned in my question. I've gotten to the point where I couldn't go without it any more, so I've implemented the simplest possible version. If it turns out that there is another implementation out there, then I'd delete mine and use it. Especially if it has tests.

Files in a hpk archive can be fragmented across the archive.
And yes, the FragmentedReader reads the file's fragments in order.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.