How to turn a Vec<Iterable<T>> into an Iterable<Vec<T>>

help
1

Hello

I am reading n multiple files (n chosen at runtime) in parallel, and thus I have a Vec<std::io::BufReader> with n elements.

I would like to iterate over all files in parallel, ie. obtaining an iterator on a Vec<Byte> (Byte == u8).

For example if I am reading a first stream containing “fool” and a second stream containing “bar”, I would like that my Iterator next method returns in sequence:

  • Some(vec!['f', 'b'])
  • Some(vec!['o', 'a'])
  • Some(vec!['o', 'r'])
  • None

I am not sure how to do that, do you have a suggestion?

2

What should happen if the streams are of different length?

1 Like
3

That’s a good question. I would like that next() on the final iterator returns None as soon as at least one input returns None. I have edited the initial question to make it more clear. Thanks.

4

In that case, your foo-bar example should perhaps be

  • [f, b]
  • [o, a]
  • [o, r]

Without the final [None, None]? And in general you wouldn’t need the Option but could expect an “Iterable<Vec<T>>”?

5

You are right. I edited the initial question.

6

I cannot find a fitting existing iterator adapter right now, even in itertools, so one approach would be to write your own.

7

Thank you :slight_smile:

8 
pub fn zip_many<I>(iterators: I) -> ZipMany<I::Item>
where
    I: IntoIterator,
    I::Item: Iterator,
{
    ZipMany {
        iterators: iterators.into_iter().collect(),
    }
}

pub struct ZipMany<I> {
    iterators: Vec<I>,
}

impl<I: Iterator> Iterator for ZipMany<I> {
    type Item = Vec<I::Item>;
    fn next(&mut self) -> Option<Vec<I::Item>> {
        self.iterators.iter_mut().map(<_>::next).collect()
    }
}

use std::io::{self, Read};
fn how_to_use(x: Vec<impl Read>) {
    let foo = zip_many(x.into_iter().map(<_>::bytes));
    for x in foo {
        // foo is an iterator of Vec<io::Result<u8>>
        let _: Vec<io::Result<u8>> = x;
    }
}


pub fn zip_many_results<I>(iterators: I) -> ZipManyResults<I::Item>
where
    I: IntoIterator,
    I::Item: Iterator,
{
    ZipManyResults {
        iterators: iterators.into_iter().collect(),
    }
}

pub struct ZipManyResults<I> {
    iterators: Vec<I>,
}

impl<T, E, I: Iterator<Item = Result<T, E>>> Iterator for ZipManyResults<I> {
    type Item = Result<Vec<T>, E>;
    fn next(&mut self) -> Option<Result<Vec<T>, E>> {
        self.iterators.iter_mut().map(<_>::next).collect()
    }
}

fn how_to_use2(x: Vec<impl Read>) -> io::Result<()> {
    let foo = zip_many_results(x.into_iter().map(<_>::bytes));
    for x in foo {
        // foo is an iterator of io::Result<Vec<u8>>
        let _: io::Result<Vec<u8>> = x;
        let y: Vec<u8> = x?; // unpack the result :-)
    }
    Ok(())
}

Rust Playground

1 Like
9

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.