Rayon sequential iterator

code review
1

I'm trying to understand the issues in rayon#210, why would something like the following snippet not be appropriate for rayon ? I'd really appreciate some re-explanations of the issues raised in that issue thread.

use crossbeam_channel::IntoIter;
use rayon::iter::ParallelIterator;

/// Unordered sequential iterator
pub trait IntoSequentialIterator {
    type Item;
    fn into_seq_iter(self) -> IntoIter<Self::Item>;
}

impl<I> IntoSequentialIterator for I
where
    I: ParallelIterator,
{
    type Item = <I as ParallelIterator>::Item;

    fn into_seq_iter(self) -> IntoIter<Self::Item> {
        rayon::scope(|scope| {
            let (sender, receiver) = crossbeam_channel::unbounded();
            scope.spawn(move |_| self.for_each(|item| sender.send(item).unwrap()));
            receiver.into_iter()
        })
    }
}

Edit: that version is not quite what you'd want (it essentially collects everything in the channel) but it should get the point accross.

2

I think a slight problem with this is unbounded memory usage. Ideally you'd want the sequential iterator to act as a backpressure for starting new tasks, so it would only buffer number of items equal to number of threads in the threadpool.

but if you just change it to channel::bounded(n), you may get deadlocks, since blocked senders will also prevent rayon threads from stealing other work.

3

For that particular example using a bounded channel would (almost) always deadlock (unless they are strictly less items than the channel capacity), since it can only return the IntoIter of the channel once both scopes (the inner and outer) are finished.

But there should be a way to expose that kind of API ? and that's more my point. Passing to and from a parallel iterator to a sequential one. C# LINQ has a way of expressing that, so I guess this isn't a theoritical limit, but more than there isn't a good implementatoin of that (and I can't figure one out right now?).

4

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.