Read a stream from and in an iterator

help
1

Hello,

I wrote a lib that split a stream reader (called buf_read_splitter)

The basic usage needs too much lines of code for a so simple purpose. For example, to count parts separates by "<SEP>" and total number of characters (excluding separators size) :

    /*** Declarations ***/
    let mut stream = ....

    let separator = "<SEP>";

    let mut reader = BufReadSplitter::new(
        &mut stream,
        SimpleMatcher::new(separator.as_bytes()),
        Options::default(),
    );

    let mut buf = vec![0u8; buf_size];

    let mut nb_part_found = 0usize;
    let mut nb_chars = 0usize;

    /*** Loop ***/
    while {
        let sz = reader.read(&mut buf).unwrap();
        if sz > 0 {
            nb_chars += sz;
            true
        } else {
            nb_part_found += 1;
            match reader.next_part().unwrap() {
                //Pass to the next part of the buffer
                Some(_) => true, //There's a next part
                None => false,   //End of the stream
            }
        }
    } {}

I'd like to offer the possibility of an iterator, for example this code below :

    ...
   
    /*** Loop ***/
    for  part in reader.iter() {
        let sz = part.read(&mut buf).unwrap();
        if sz > 0 {
            nb_chars += sz;
            true
        } else {
            nb_part_found += 1;
        }
    }

I don't find any solution because of the borrow checker between the stream owns by reader and (so) can't be own by part.

Do you have any idea of how to do something like that ?

2

I'm not 100% sure this is what you want, but maybe you will be interested in this kind of API. Note that I didn't implement it, so there is some chance that there are still borrow checker hazards waiting down the line, but I don't see any now.

use std::io::{self, BufRead};

pub trait Matcher {
    /* ... */
}

pub trait BufReadUntilPatExt: BufRead {
    fn read_until_pat<'io, 'buf, M: Matcher>(
        &'io mut self,
        pattern: M,
        buf: &'buf mut Vec<u8>,
    ) -> BufReadUntilPat<'io, 'buf, Self, M>
    where
        Self: Sized;
}

impl<T> BufReadUntilPatExt for T
where
    T: BufRead,
{
    fn read_until_pat<'io, 'buf, M: Matcher>(
        &'io mut self,
        pattern: M,
        buf: &'buf mut Vec<u8>,
    ) -> BufReadUntilPat<'io, 'buf, Self, M>
    where
        Self: Sized,
    {
        BufReadUntilPat {
            io_buf: self,
            matcher: pattern,
            buf,
        }
    }
}

pub struct BufReadUntilPat<'io, 'buf, B, M> {
    io_buf: &'io mut B,
    matcher: M,
    buf: &'buf mut Vec<u8>,
}

// Option 1. Iterator, which requires copying data.
impl<'io, 'buf, B, M> Iterator for BufReadUntilPat<'io, 'buf, B, M>
where
    B: BufRead,
    M: Matcher,
{
    type Item = io::Result<Box<[u8]>>;

    fn next(&mut self) -> Option<Self::Item> {
        let chunk = match self.lend_next() {
            Ok(chunk) => chunk,
            Err(error) => return Some(Err(error)),
        };

        if chunk.is_empty() {
            return None;
        }

        Some(Ok(Box::from(chunk)))
    }
}

// Option 2. LendingIterator like method. Borrows data from `self.buf`.
impl<'io, 'buf, B, M> BufReadUntilPat<'io, 'buf, B, M>
where
    B: BufRead,
    M: Matcher,
{
    pub fn lend_next<'loan>(&'loan mut self) -> io::Result<&'loan [u8]> {
        // See default implementation of BufRead::read_until and adjust.

        // Proof that lifetimes work.
        Ok(&self.buf[..0])
    }
}

fn _test<B: BufRead>(mut buf: B) {
    struct Unit;
    impl Matcher for Unit {}

    for _segment in buf.read_until_pat(Unit, &mut vec![]) {}

    let mut b = vec![];
    let mut lender = buf.read_until_pat(Unit, &mut b);
    while let Ok(segment) = lender.lend_next() {
        if segment.is_empty() {
            break;
        }
    }
}
3

Looks like you can avoid returning something borrowed if you have the iterator do the reading for you.

pub enum IterItem {
    Read(NonZeroUsize),
    NextPart,
}

pub struct Iter<'m, 'a, T> {
    brs: &'m mut BufReadSplitter<'a, T>,
    buf: &'m mut [u8],
    // ...
}

impl<T> Iterator for Iter<'_, '_, T> {
    type Item = Result<IterItem, Box<dyn Error>>;
    fn next(&mut self) -> Option<Self::Item> { ... }
}

    for part in bsr.read_iter(&mut buf) {
        match part.unwrap() {
            IterItem::Read(sz) => nb_chars += sz.get(),
            IterItem::NextPart => nb_part_found += 1,
        }
    }

Untested:

5

Ha sorry, I have to amend the what-I-want (with my apologies). In fact it's a loop in a loop :

  • the first level read through the " part "
  • the second level read the buffer of each part

It's the difficulty : the reader.iter() borrow part, so part or reader can't be use inside the second level :

    ...
   
    /*** Loop ***/
    for  part in reader.iter() { // <-- borrow "reader"
        loop {
            let sz = part.read(&mut buf).unwrap();
            if sz > 0 {
                nb_chars += sz;
            } else {
                break; 
            }
        }
        nb_part_found += 1;
    }

Maybe there is a fundamental design issue in the library (?)

6

I see (I think). Well, you could use a lending iterator pattern, where you use while let Some(_) instead of for. But it's correct that needing to use &mut BufReadSplitter in the next method and in the loop prevents any meaningful Iterator implementation.

I must admit I haven't taken the time to truly understand the library. But I do find the next_part dance a bit odd.

Maybe BufReadSplitter could return (lending iterator style) a BufPartReader that acts like a typical reader? This feels a little like the contrast between

  • A unfused iterator that can start returning Some again after None
    • And you handle some double loop yourself until you see two Nones or something
  • A fused iterator than returns other fused iterators
// Brainstorming
let brs = BufReadSplitter::new(...); // <-- Doesn't implement Read
while let Some(part) = brs.next_part() {
    //        vvvvvvvvvvvvv Does implement Read
    let part: BufPartReader<'_, '_, _> = part.unwrap();
    // Do your reading of a single part in a loop here
}

Similar to the playground, but there's a new type with a dedicated "act more like a normal reader" roll, instead of using BufReadSplitter for everything.

7

Yes, it's odd for the caller's point of view because : the concept is, to split one Stream into several (split by some predicate), iterate over each one, and in each iteration read the stream as usual (by a buffer).

But under the hood there's only one Stream that is shared between the Split-Stream-Iteration and the Read-One-Stream (and that's where the ownership problem originates)

You are right, it reduces the number of lines to code, it's clearer. Not as sexy as an iterator, but a good simplification that work fine. Thanks @quinedot !