Process stream in parallel, keep order

help
1

Hi all!

I'm trying to find, or to create, a helper tool which basically operates on a stream of data, processes the data in parallel (CPU intensive and slow operation), and output the result of the processed data.

Important is that the output data is kept in order of first in (as a FIFO), and that input is blocked if the current number of processed elements is too long.

Rayon seemed almost like a good choice but couldn't understand how to make it operate on a stream where not all elements are present when starting the processing.

My current solution involves some MPSC channels and a thread pool, but feels sub-optimal and can't handle back pressure right now.

Is there some crate out there like this, or any proposals of how to implement this in a Rusty way? :slight_smile:

1 Like
2

Rayon has par_bridge for arbitrary iterators, but that does not preserve order. You could enumerate your input and sort the result.

The par_map crate may be more directly useful to you, as it does give you an iterator output which preserves the input iterator's order.

1 Like