New crate: olio

https://crates.io/crates/olio

Provides some I/O-related utilities complimenting std::io and std::fs. I split these more general utilities out of recent work on body-image. Includes:

GatheringReader presents a continuous Read interface over N non-contiguous byte buffers, and is used for the scattered Ram variant of BodyImage. This is more efficient than the current implementation ofstd::io::Cursor::chain for many reads over many buffers. See the associated benchmark comparison.

The ReadPos and ReadSlice types support multiple independent instance positions over a shared File reference, without needing a path to open an independent new File instance. Thus they are compatible with "unnamed" (not linked) temporary files (tempfile @stebalien) , and can reduce the number of necessary file handles. Note that unix dup/dup2 and the standard File::try_clone do not provide independent file positions.

I couldn't find anything comparable to GatheringReader, but after an initial prototype, I did find and considered using the positioned-io crate for the latter. This crate has broader application including positioned writes and much more flexibility. However, I couldn't quite achieve what I needed with positioned-io—a Read-only slice-type facade using a shared (Arc<File>) reference (olio::fs::rc::ReadSlice).

If @vasi was interested to comment here or on github at the possibility of extending positioned-io for these use-case variants, I'm potentially interested in contributing this and some more time to adapt it.

8 Likes

It kind of cool when benchmark results dramatically improve between rust nightly releases:

rustc 1.27.0-nightly (bd40cbbe1 2018-04-14):

test gather_chained_cursors   ... bench:     558,877 ns/iter (+/- 90,532)
test gather_reader            ... bench:      63,256 ns/iter (+/- 2,294)
test gather_upfront           ... bench:      64,078 ns/iter (+/- 14,701)
test gather_upfront_read_only ... bench:      40,490 ns/iter (+/- 3,578)

rustc 1.27.0-nightly (acd3871ba 2018-05-10):

test gather_chained_cursors   ... bench:     540,762 ns/iter (+/- 11,658)
test gather_reader            ... bench:      34,323 ns/iter (+/- 7,333)
test gather_upfront           ... bench:      45,337 ns/iter (+/- 1,058)
test gather_upfront_read_only ... bench:      24,184 ns/iter (+/- 901)
2 Likes

Would appreciate any feedback on this potential use of the Barrow type, to expand use cases for ReadPos/Slice beyond prior release:

https://github.com/dekellum/olio/pull/1

And here is my offer to positioned-io:

https://github.com/vasi/positioned-io/issues/3