Is there some way to convert rayon parallel iterator back to sequential iterator?

[hombit]

I have some iterator that produces a large amount of data (not fitting into the memory), which I'd like to process in parallel manner and write each result into the same file. I found rayon to be very cool tool to do parallel things, but, obviously, I cannot use mutable buffer in the parallel region, and I wouldn't .collect() results because I cannot allocate such a large amount of memory. Is there any possibility to back my application flow to sequential execution?

(Broken) code snippet to explain my problem:

use std::fs::{File};
use std::io::{BufWriter, Write};
use rayon::prelude::*;

fn main() {
    let file = File::create("/tmp/data.txt").expect("Cannot create file");
    let mut buf = BufWriter::new(file);
    ["hello", "world"].par_iter()
        .map(|s| String::from(*s) + "_")
        // Back to sequential execution somehow
        .for_each(move |s| {
            buf.write(s.as_bytes()).expect("Cannot write to file");
        });
}

Sorry if this question is discussed some basic things, I'm newbie in both Rust and parallel programming.

[daboross]

FR: Parallel to sequential iterator · Issue #210 · rayon-rs/rayon · GitHub may be helpful!

From what I gathered looking at the issue, I don't think this functionality is natively available in rayon.

One thing which should be fairly simple is to create a mpsc::channel, have the parallel iterator "end" by sending all the results into the channel, and then process the results by reading them from the channel. As long as either the "reading from the channel" or "sending to the channel" is done in a separate thread, it should still work in parallel.

If you use a sync channel instead, it'll limit the number of items in the buffer, so if writing to the file ends up being the bottleneck, you'll have back-pressure and stop processing so that you don't end up filling up memory.

[hombit]

Thank you! I'll try to implement it with channels