[Solved] Rayon only running 24 threads; CPU only at 5%

1

  1. I'm using blahblah.par_iter().map(....).collect::<Vec<_>>();

  2. My machine has 24 cores. Ryaon is running 24 threads. However, according to 'top', my CPU is only at 5%.

  3. What's going on? I thought rayon was supposed to maximally utilize my machine's resources. (All the threads are constantly doing hashmap lookups, so I think there's lots of delys due to memory latency -- in which case I'm expecting more threads to get fired up.)

1 Like
2

How often does it run? And how long does it take to run? There might just not be enough work to actually need 24 threads, and therefore the work is basically done before it can send it out to the respective threads...

3

There are 156 "jobs". Only 24 runs at any given time. It takes ~10 minutes for all 156 to finish.

4

Each job is take this Vec convert it to Vec by looking it up in this HashMap.

5

Okay. A quick lookup found this, and I would use this code:

std::env::set_var("RAYON_NUM_THREADS", "50");
//Do your work
std::env::set_var("RAYON_NUM_THREADS", "");

or similar, to not break other code that works with rayon
When I was experimenting with rust on android, I would do this with RUST_BACKTRACE like so:

fn main() {
    std::env::set_var("RUST_BACKTRACE", "1");
}
1 Like
6

@OptimisticPeach : This explains it. Thanks!

1 Like
7

Also, in the case you want more configurability then try to make your own thread pool.
Link

8

I think, at thtak point, I'll just use crossbeam :slight_smile:

1 Like