How can I use rayon to concurrently collect function results

Hi, all guys.

I have a function search_core(start, end) which returns a Result<Vec<SearchResult>, SearchError>, I wanna spawn multiple search_core() with different start, end inputs, so I can perform concurrent search and combine all the Vec<SearchResult> into one and return to user.

I'm new to Rust and new to rayon, anyone can help with this?

  • You prepare a Vec<(start, end)>.
  • You then .into_par_iter().map(|(s,e)| search_core(s, e) ).collect();
  • You now have a Iter<Result<Vec<SearchResult>, SearchError>>.
  • Now you .fold() or .reduce() it to a single Vec<SearchResult>.
1 Like

Is there any performance degradation for the chain call of map -> collect -> fold/reduce?

Or, is there any better way to improve the overall performance?

Also, does the par_iter() seem to have no fine-grained thread pool control instead of installing a global thread pool beforehand?

Is there any performance degradation [...]

Compared to which alternative?

Or, is there any better way to improve the overall performance?

We don't know what search_core() does.

Also, does the par_iter() seem to have no fine-grained thread pool control instead of installing a global thread pool beforehand?

Yes. It seems so. For fine grained solution you'd probably need some other crate (tokio has such functionality, FuturesUnordered) or roll your own executor.

Check out ThreadPool::install (the name is imo misleading, as it allows you to use a local ThreadPool temporarily, without "installing" anything globally).

The install seems a blocking method?

Yeah, although I don't see how's that a problem? You want to put your iterator chain inside it (.install(|| ... .collect())), and collect is already blocking.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.