Use rayon with itertools

help
1

I try to use rayon.ParallelBridge with iterator produced by itertoos.Itertools.group_by():

use rayon::prelude::*;
use itertools::Itertools;

fn main() {
    let s: i32 = [0, 1, 2, -3, -4, -5, 6, 7, 8, 9, -10]
        .iter()
        .group_by(|&x| *x >= 0)
        .into_iter()
        .par_bridge()
        .map(|(_key, group)| group.sum::<i32>())
        .sum();
    println!("{}", s);
}

error[E0599]: no method named `par_bridge` found for type `itertools::groupbylazy::Groups<'_, bool, std::slice::Iter<'_, {integer}>, [closure@src/main.rs:7:19: 7:31]>` in the current scope
 --> src/main.rs:9:10
  |
9 |         .par_bridge()
  |          ^^^^^^^^^^
  |
  = note: the method `par_bridge` exists but the following trait bounds were not satisfied:
          `&itertools::groupbylazy::Groups<'_, bool, std::slice::Iter<'_, {integer}>, [closure@src/main.rs:7:19: 7:31]> : rayon::iter::par_bridge::ParallelBridge`
          `&mut itertools::groupbylazy::Groups<'_, bool, std::slice::Iter<'_, {integer}>, [closure@src/main.rs:7:19: 7:31]> : rayon::iter::par_bridge::ParallelBridge`
          `itertools::groupbylazy::Groups<'_, bool, std::slice::Iter<'_, {integer}>, [closure@src/main.rs:7:19: 7:31]> : rayon::iter::par_bridge::ParallelBridge`

Can you help, what particular bound isn't satisfied? into_iter() should return Iterator while rayon.ParallelBridge is implemented for Iterator

2

Because Group is not Send.

1 Like
3

More accurately, it's implemented for any type that implements Iterator and Send and where the iterated items implement Send:

impl<T: Iterator + Send> ParallelBridge for T
where
    T::Item: Send,

You can test if the iterator implements Send yourself:

use itertools::Itertools;

fn main() {
    let tmp = [0, 1, 2, -3, -4, -5, 6, 7, 8, 9, -10]
        .iter()
        .group_by(|&x| *x >= 0)
        .into_iter();
        
    fn is_send<T: Send>(_: &T) {}
    is_send(&tmp);
}

error[E0277]: `std::cell::RefCell<itertools::groupbylazy::GroupInner<bool, std::slice::Iter<'_, {integer}>, [closure@src/main.rs:6:19: 6:31]>>` cannot be shared between threads safely
  --> src/main.rs:10:5
   |
10 |     is_send(&tmp);
   |     ^^^^^^^ `std::cell::RefCell<itertools::groupbylazy::GroupInner<bool, std::slice::Iter<'_, {integer}>, [closure@src/main.rs:6:19: 6:31]>>` cannot be shared between threads safely
   |
   = help: within `itertools::groupbylazy::GroupBy<bool, std::slice::Iter<'_, {integer}>, [closure@src/main.rs:6:19: 6:31]>`, the trait `std::marker::Sync` is not implemented for `std::cell::RefCell<itertools::groupbylazy::GroupInner<bool, std::slice::Iter<'_, {integer}>, [closure@src/main.rs:6:19: 6:31]>>`
   = note: required because it appears within the type `itertools::groupbylazy::GroupBy<bool, std::slice::Iter<'_, {integer}>, [closure@src/main.rs:6:19: 6:31]>`
   = note: required because of the requirements on the impl of `std::marker::Send` for `&itertools::groupbylazy::GroupBy<bool, std::slice::Iter<'_, {integer}>, [closure@src/main.rs:6:19: 6:31]>`
   = note: required because it appears within the type `itertools::groupbylazy::Groups<'_, bool, std::slice::Iter<'_, {integer}>, [closure@src/main.rs:6:19: 6:31]>`
note: required by `main::is_send`
  --> src/main.rs:9:5
   |
9  |     fn is_send<T: Send>(_: &T) {}
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^
1 Like
4

Thank you!

I understand that Group cannot be Send by design, each group is a part of the same iterator. But I still doesn't understand why iterator itself should be Send?

5

The iterator needs to be Send because rayon's threads are going to read from it (through a Mutex).

2 Likes
6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.