Simple_line, a fast&secure alternative to std::io::BufReader::

mineichen · January 8, 2021, 1:19pm

I'm about to release my first version of simple_line, a crate to iterate over lines of a std::io::Read. But wait: This is what std::io::BufReader does which is part of the standard library. I've also written about why I did so.

github.com

mineichen/simple_lines/blob/main/devto.md


It was great fun solving last year's [Advent of Code](https://adventofcode.com/2020) using Rust. 
The problems were all approachable and got me thinking about new algorithms to solve them. A small part of the resulting code ended up in a "utils" crate to be accessible from every sub project in the workspace. The utils-crate included a struct called LineReaderIterator with the following signature:
```rust
pub struct LineReaderIterator<T, TFn: FnMut(&str) -> Result<T, Error>, TRead: Read> {
    reader: BufReader<TRead>,
    buffer: String,
    mapper: TFn,
}
```

But why would you write your own iterator when there is `std::io::BufRead::lines()` in the standard library? When I looked at the lines()-signature, it resulted in a minor hickup: It returns a fresh `String` for each line instead of reusing a common buffer after the line was processed. Of course this never really impacted performance with such small input files, but as the entire AoC is about fun, I wanted to do that without this additional allocation.

The above iterator was able to parse each line into an appropriate struct. However, it wasn't perfect. When the first puzzle with multiple input sections arrived, I ended up using `std::BufRead::lines()` again, because there was no way to stop parsing and process the rest of the lines with another reducer. I thought that there must be a way to implement an `Iterator<Item=&mut str>`. After all, `slice::IterMut` also returns mutable references to it's data with an iterator. 

Unfortunately it turned out you can't, because the signature of iterator makes it impossible to return a reference which only lives until the next call to `next()`.
```rust
next(&mut self) -> Option<Self::Item>;
```

This file has been truncated. show original

I'd very much appreciate if anyone could find some time to review the article and give me feedback about it and the library before I release it as my first article on dev.to and first library to crates.io. Best thanks in advance

quinedot · January 8, 2021, 10:43pm

A lending iterator is in fact the motivating example of RFC 1598 (GATs).

Don't get confused by the name in the example; the proposed async Streams are something else. However, there is a discussion in that RFC about how lending and non-lending Streams may interact, and some coherence issues around it. The same considerations apply to lending and non-lending Iterators. (TLDR: you can blanket impl the lending version for the non-lending version, but not without coherence hurdles.)

mineichen · January 16, 2021, 2:44pm

Sorry for my delay. I complemented my post with your notes and referred to you in my final version on Writing a better Line Iterator in Rust - DEV Community. Thank you very much for your participation!

system · April 16, 2021, 2:44pm

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.