How can I parallelize this with rayon and return Err on first failure?

help
1

I've got the following code

pub fn list_devices(&self) -> Result<Vec<MyDevice>, MyError> {
    let mut results = Vec::new();

    for driver in &self.drivers {
        let serial_numbers = driver.list_serial_numbers()?;

        for serial in serial_numbers {
            results.push(MyDevice::new(driver.clone(), &serial)?);
        }
    }

    Ok(results)
}

However, both driver.list_serial_numbers() and MyDevice::new() can take seconds to return and doing all this long-running work in sequence can take a long time. I'd like to take advantage of rayons par_iter to do both iterations in parallel, but as per the above code, I'd like to return an error if any result in failure.

So far, I've come up with the following code which works but does not handle errors correctly. I need to remove the usages of unwrap and I know that flatten() is also ignoring errors so that needs replacing with something else too...

pub fn list_devices(&self) -> Result<Vec<MyDevice, MyError>> {
    Ok(self
        .drivers
        .par_iter()
        .map(|driver| -> Vec<MyDevice> {
            driver
                .list_serial_numbers()
                .unwrap()
                .par_iter()
                .map(|serial| MyDevice::new(driver.clone(), serial).unwrap())
                .collect()
        })
        .flatten()
        .collect())
}
2

Without the flatten, you can collect Result<T, E> items directly into a Result<C, E> collection, and it will stop at the first error E encountered (in arbitrary parallel order).

edit: I missed the error from list_serial_numbers too -- I'll think about that...

3

How about this?

pub fn list_devices(&self) -> Result<Vec<MyDevice, MyError>> {
    self
        .drivers
        .par_iter()
        .flat_map(|driver| -> Vec<Result<MyDevice, MyError>> {
            match driver.list_serial_numbers() {
                error @ Err(_) => vec![error],
                Ok(serials) => serials.par_iter()
                    .map(|serial| MyDevice::new(driver.clone(), serial))
                    .collect(),
            }
        })
        .collect()
}
3 Likes
4

Will have a better look when I get back to my desk.

MyDevice.new() also returns a Result that needs catering for!

I've found that rayon has a load of try_ methods that I need to investigate too.

5

Oh, I meant to remove that unwrap() on new() too, so we're collecting all Results as-is. Edited, but I can't exactly test this without mocking up the rest of the code.

6

Thanks, your answer was most of the way there and helped tremendously!

I got the following error:

  |
  |    error @ Err(_) => vec![error],
  |                           ^^^^^ expected struct `MyDevice`, found struct `Vec`
  |
    = note: expected type `Result<MyDevice, _>`
               found type `Result<<String>, _>`

But it works with Err(error) => vec![Err(error)].

Iterators in Rust are made trickier because many of the methods are powerful and can perform many functions depending on the input!

1 Like
7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.