Parallel work collected sequentially

help
2

The general request can be found in rayon#210, but we don't have a great solution yet.

The workaround that I often suggest is to batch your work in manageable chunks. i.e. do the parallel map(m).collect() on just the first part of your input, then serially reduce that, then do another parallel section, reduce, etc. etc.

let mut result = init();
for chunk in input.chunks(N) {
    let buffer = chunk.par_iter().map(m).collect();
    result = reduce(result, buffer);
}

You might even do the reduction and next part in parallel so you don't have serialization stalls, like:

// do the first chunk on its own
let mut buffer = input[..N].par_iter().map(m).collect();

let mut result = init();
for chunk in input[N..].chunks(N) {
    // reduce the prior chunk in order, while we also process the next chunk
    let (r, b) = rayon::join(|| reduce(result, buffer),
                             chunk.par_iter().map(m).collect());
    result = r;
    buffer = b;
}

// reduce the final chunk
reduce(result, buffer)

I wonder if the library could encapsulate some patterns like this, but it may be too domain specific... not sure.