What is needed to have a custom iterator interop with rayon?

help
1

Hello!

I have an iterator similar to std::str::Lines that returns slices into a collection where each slice is one page of the entire collection.

pub trait Pages<T: HasRect> {
    fn pages(&self) -> PagesIter<'_, T>;
}

impl<T: HasRect> Pages<T> for [T] {
    fn pages(&self) -> PagesIter<'_, T> {
        PagesIter::new(self)
    }
}

impl<T: HasRect> Pages<T> for &[T] {
    fn pages(&self) -> PagesIter<'_, T> {
        PagesIter::new(self)
    }
}

#[derive(Debug, Clone)]
/// An iterator over pages in a collection
#[must_use = "iterators are lazy and do nothing unless consumed"]
pub struct PagesIter<'a, T: HasRect> {
    collection: &'a [T],
    i: usize,
}

impl<'a, T: HasRect> Iterator for PagesIter<'a, T> {
    type Item = &'a [T];
    //...

I tried par_bridge() but it needs me to follow the compiler errors (needs Send, Sized, needs Clone) etc, and I'm getting a little lost here, I tried pre-collecting into a Vec of Vec, but then the T is still a problem for Rayon.

Should PagesIter impl ParallelIterator?

Perhaps someone could guide me here, or point me to a blog or resource?

Thanks!

Edit: crap, it works: Rust Playground
Have to go figure out why it's complaining my in project then...

2

Right, this is the error I'm having concretely in my project (by trying to add Send bounds everywhere)

error[E0599]: the method `par_bridge` exists for struct `PagesIter<'_, T>`, but its trait bounds were not satisfied
  --> my_project.rs:18:10
   |
18 |         .par_bridge()
   |          ^^^^^^^^^^ method cannot be called on `PagesIter<'_, T>` due to unsatisfied trait bounds
   |
  ::: crate/src/pages.rs:22:1
   |
22 | pub struct PagesIter<'a, T: HasRect + Send> {
   | -------------------------------------------
   | |
   | method `par_bridge` not found for this struct
   | doesn't satisfy `PagesIter<'_, T>: Send`
   | doesn't satisfy `PagesIter<'_, T>: rayon::iter::ParallelBridge`
   |
   = note: the following trait bounds were not satisfied:
           `PagesIter<'_, T>: Send`
           which is required by `PagesIter<'_, T>: rayon::iter::ParallelBridge`
           `&[T]: Send`
           which is required by `PagesIter<'_, T>: rayon::iter::ParallelBridge`
           `&PagesIter<'_, T>: Iterator`
           which is required by `&PagesIter<'_, T>: rayon::iter::ParallelBridge`
           `&PagesIter<'_, T>: Send`
           which is required by `&PagesIter<'_, T>: rayon::iter::ParallelBridge`
           `&mut PagesIter<'_, T>: Send`
           which is required by `&mut PagesIter<'_, T>: rayon::iter::ParallelBridge`
           `&[T]: Send`
           which is required by `&mut PagesIter<'_, T>: rayon::iter::ParallelBridge`
note: the trait `Send` must be implemented
  --> /home/arif/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/marker.rs:38:1
   |
38 | pub unsafe auto trait Send {
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^

You can see I have restricted T to be Send so why cant the compiler infer my iterator is Send?

And why does the playground work, without specifying Send?

3

What's the larger context of the error site? E.g. function declaration, impl header.

4

I think you have it all already

Here's the callsite:

pub fn determine_clusters<T: HasRect>(collection: &[T]) -> Vec<Vec<&T>> {
    collection
        .pages() // returns an iterator over `&[T]`
        .par_bridge() // errors here
        //...

the impls are:

impl<T: HasRect> Pages<T> for [T] {
    fn pages(&self) -> PagesIter<'_, T> {
        PagesIter::new(self)
    }
}

impl<T: HasRect> Pages<T> for &[T] {
    fn pages(&self) -> PagesIter<'_, T> {
        PagesIter::new(self)
    }
}

with:

#[derive(Debug, Clone)]
/// An iterator over pages in a collection
#[must_use = "iterators are lazy and do nothing unless consumed"]
pub struct PagesIter<'a, T: HasRect> {
    collection: &'a [T],
    i: usize,
}

impl<'a, T: HasRect> PagesIter<'a, T> {
    pub fn new(collection: &'a [T]) -> Self {
        PagesIter { collection, i: 0 }
    }
}

the iterator is:

impl<'a, T: HasRect> Iterator for PagesIter<'a, T> {
    type Item = &'a [T];
5

Interesting, even with unsafe impl<T: HasRect> Send for PagesIter<'_, T> {} I still get an error, and it is this:

error[E0599]: the method `par_bridge` exists for struct `PagesIter<'_, T>`, but its trait bounds were not satisfied
  --> myproject/foo.rs:13:10
   |
13 |         .par_bridge()
   |          ^^^^^^^^^^ method cannot be called on `PagesIter<'_, T>` due to unsatisfied trait bounds
   |
  ::: myproject/bar.rs:22:1
   |
22 | pub struct PagesIter<'a, T: HasRect> {
   | ------------------------------------
   | |
   | method `par_bridge` not found for this struct
   | doesn't satisfy `PagesIter<'_, T>: rayon::iter::ParallelBridge`
   |
   = note: the following trait bounds were not satisfied:
           `&[T]: Send`
           which is required by `PagesIter<'_, T>: rayon::iter::ParallelBridge`
           `&PagesIter<'_, T>: Iterator`
           which is required by `&PagesIter<'_, T>: rayon::iter::ParallelBridge`
           `&PagesIter<'_, T>: Send`
           which is required by `&PagesIter<'_, T>: rayon::iter::ParallelBridge`
           `&[T]: Send`
           which is required by `&mut PagesIter<'_, T>: rayon::iter::ParallelBridge`
6

I was looking for information to reproduce your error. You just used main in your playground (no generics) and there were no Send bounds anywhere in the playground or in your OP, but the error you posted right above my first reply has them. There are none in this post either.

Anyway, my best guess is that you need to add

pub fn determine_clusters<T: HasRect + Sync>(collection: &[T]) -> Vec<Vec<&T>>
//                                   ^^^^^^
2 Likes
7

Oh man, i think you're absolutely right! Unconfirmed in my real project because I'm on mobile, but this playground seems to prove it!

Thank you!!

Should i create an issue somewhere about an unhelpful error message?

8

I couldn't find one off-hand, so I think it makes sense to do so. There's a diagnostic template here.

A very brief summary is "suggest T: Sync when &T: Send is required".

9

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.