Library for buffered Iterator?

[dctremblay]

Is there a library which :

takes input : Iterator and n buffer limit entries
output : Iterator of buffered queued data which continues to iterate in background

Or this isn't good practice and/or there is a more better way?

Thanks

[drewkett]

For IO, this is effectively what std::io::BufReader does. The purpose of that is to minimize the number of system calls.

What’s your use case/design goal with buffering?

[Michael-F-Bryan]

@dctremblay

which continues to iterate in background

This point will probably cause you trouble. What do you mean by "continues to iterate in background"? Do you want to spin up a new thread that does the iteration and sends the results back to the main thread? Otherwise the only time your buffered iterator will get to execute code is when the next() method is called, in which case buffering doesn't give you much.

Here's a simple version using channels and a background thread. if performance is an issue for you, either look into using a threadpool (to ammortize the cost of creating threads) or async IO.

use std::sync::mpsc::self;

pub fn buffered<I>(iter: I, buffer_size: usize) -> impl Iterator<Item = I::Item>
where
    I: IntoIterator,
    I::IntoIter: Send + 'static,
    I::Item: Send + 'static,
{
    let iter = iter.into_iter();
    let (sender, items) = mpsc::sync_channel(buffer_size);

    std::thread::spawn(move || {
        for item in iter {
            if sender.send(item).is_err() {
                // the BufferedIterator was dropped and we no longer care about the iterator
                return;
            }
        }
    });

    items.into_iter()
}