Confusion about ```into_par_iter```

Hi all,

I'm learning rust and still a bit confused about into_par_iter function.

I have an matrix (named matrix) of type ndarray::Array2, and I initialize it with all zeros. I also have an another matrix (named another_matrix) of the same type and same shape. My purpose is, iterate over matrix, assign the corresponding row in another_matrix to matrix, but since matrix is quite huge, I want to use multi-threading. Therefore, I'm using into_par_iter() to parallel the process.

Here is my code:

env_logger::init();

        ThreadPoolBuilder::new()
            .num_threads(self.n_threads)
            .build_global()
            .unwrap();

// some more irrelevant code here...

let mut matix= Array2::zeros(storage.shape());
for (idx, row) in matix.outer_iter_mut().into_par_iter().enumerate() {
        row.assign(&anothor_matrix.row(idx));
}

But the compiler says:

rayon::iter::enumerate::Enumerate<ndarray::parallel::par::Parallel<ndarray::iterators::AxisIterMut<'_, _, ndarray::dimension::dim::Dim<[usize; 1]>>>>` is not an iterator

How do I fix this?

Thank you!

You need to use for_each not a for loop. For loops only accept normal iterators, not rayon's parallel iterators.

matix.outer_iter_mut()
    .into_par_iter()
    .enumerate()
    .for_each(|(idx, row)| {
        row.assign(&anothor_matrix.row(idx));
});

But this could be made a little better by instead using zip to iterate over the two matricies in lock step.

matix.outer_iter_mut()
    .into_par_iter()
    .zip(another_matrix.outer_iter())
    .for_each(|(row, other_row)| {
        row.assign(other_row);
});
3 Likes

A rayon parallel iterator is not a iterator recognized by rustc. Parallelizing this problem wont be possible, as you can only mutate a value from one thread at a time. Also moving big amount of data is likely to be memory bound, not cpu bound, so parallelizing wont help, as all threads share the same memory bandwidth.

No, that's not the issue ndarray provides parallel iterators that can iterate over the matrix on multiple threads and give you unique references to it's elements.

You can use that to initialize the matrix as shown in my example.

1 Like

Missed you second example.

You're right

My first example also works because we are only capturing the other mattrix by shared reference. So the closure doesn't mutate it's environment.

1 Like