New crate: olio


Provides some I/O-related utilities complimenting std::io and std::fs. I split these more general utilities out of recent work on body-image. Includes:

GatheringReader presents a continuous Read interface over N non-contiguous byte buffers, and is used for the scattered Ram variant of BodyImage. This is more efficient than the current implementation ofstd::io::Cursor::chain for many reads over many buffers. See the associated benchmark comparison.

The ReadPos and ReadSlice types support multiple independent instance positions over a shared File reference, without needing a path to open an independent new File instance. Thus they are compatible with “unnamed” (not linked) temporary files (tempfile @stebalien) , and can reduce the number of necessary file handles. Note that unix dup/dup2 and the standard File::try_clone do not provide independent file positions.

I couldn’t find anything comparable to GatheringReader, but after an initial prototype, I did find and considered using the positioned-io crate for the latter. This crate has broader application including positioned writes and much more flexibility. However, I couldn’t quite achieve what I needed with positioned-io—a Read-only slice-type facade using a shared (Arc<File>) reference (olio::fs::rc::ReadSlice).

If @vasi was interested to comment here or on github at the possibility of extending positioned-io for these use-case variants, I’m potentially interested in contributing this and some more time to adapt it.

Generic over trait and Borrow type, PhantomData

It kind of cool when benchmark results dramatically improve between rust nightly releases:

rustc 1.27.0-nightly (bd40cbbe1 2018-04-14):

test gather_chained_cursors   ... bench:     558,877 ns/iter (+/- 90,532)
test gather_reader            ... bench:      63,256 ns/iter (+/- 2,294)
test gather_upfront           ... bench:      64,078 ns/iter (+/- 14,701)
test gather_upfront_read_only ... bench:      40,490 ns/iter (+/- 3,578)

rustc 1.27.0-nightly (acd3871ba 2018-05-10):

test gather_chained_cursors   ... bench:     540,762 ns/iter (+/- 11,658)
test gather_reader            ... bench:      34,323 ns/iter (+/- 7,333)
test gather_upfront           ... bench:      45,337 ns/iter (+/- 1,058)
test gather_upfront_read_only ... bench:      24,184 ns/iter (+/- 901)


Would appreciate any feedback on this potential use of the Barrow type, to expand use cases for ReadPos/Slice beyond prior release:

And here is my offer to positioned-io: