Hey Rustaceans,
I am trying to do something maybe impossible with rayon: I would like to partition a parallel iterator into multiple parallel iterators, to be able to process the partitioned iterators in parallel (in different spawns).
let (itera, iterb) = chunks.par_bridge().map(intensive_task).unzip_by(|x| (x.clone(), x));
rayon::spawn({
let output = itera.map(do_something_cpu_intensive).collect();
// send the output to the main thread using a channel
});
rayon::spawn({
let output = iterb.map(do_another_intensive_thing).collect();
// send the output to the main thread using a channel
});
I already thought about using a simple crossbeam channel, however, using a crossbeam channel would not be efficient enough as the channels do not inform the rayon runtime about the fact that a task is currently waiting and can do work-stealing instead. A crossbeam channel waiting will just waste a thread. The other problem is about the crossbeam producer/send, it will not be informed of when it needs to send a new item in the iterator and will either be slow or too fast for the underlying tasks.
The other alternative we thought about was to simply collect the elements into a Vec and then simply iterate on the slice of it, once for each job. One of the problems of this solution is that we waste time by waiting for the vector to be available/collected. The other issue is that we will need to do this split/partition at multiple steps of our program.
We thought about maybe using combinations of the Cloned iterator combinator or directly cloning the original iterator, but it doesn't help as it will just do the intensive_task multiple times. We are now reading about the plumbing part of the rayon crate, we will maybe implement our own partition iterator.
In conclusion, we are trying to optimize the work-stealing of rayon, and using channels doesn't help.
Can anyone help us?