Best way to share crossbeam stealers between threads?

Background

I have a bunch of complicated tasks I want to run in parallel. Some of the tasks add more tasks to the queue, and most of the tasks involve data with non-static lifetimes.

Based on these requirements, it seems like crossbeam's scoped threads (for sending data with non-static lifetimes across threads) are a good fit, and crossbeam's deque for work-stealing queues should work for parallelizing the tasks.

Question

When setting up the work-stealing queues, I ran into a question about how best to share the stealers between threads. (The docs don't suggest a way to do it.) Here's where I've gotten:

  • The docs for Worker recommend creating one worker queue per thread. So I call scope, spawn a thread, and create a Worker queue in there. So far, so good.
  • I also need an Injector to serve as the global task queue on the main thread. That's easy enough to populate - I push tasks onto it from the main thread, and push takes a reference, so I can share that reference across threads.
  • Finally, each Worker should have a Stealer which lets it steal tasks from other threads - like in the find_task example from the docs, a function which each thread calls to find a new task, passing a &[Stealer<T>].

This is where things got more complicated. Since each thread creates its Worker in-thread, that's also where I have to create its Stealer. However, to call find_task, each thread needs to know about a slice of all the other stealers, even though each of those stealers were created on other threads.

To build this list of stealers, I set up a channel and had each thread send a message to it containing the Stealer it just created. Then the main thread gathered them all up into a Vec. This works, but then I hit a problem: how can I now give each thread access to (a slice of) the completed Vec?

They can't borrow it, since it's borrowed mutably on the main thread (to populate it), which necessarily happens after the closure which would like to borrow it. I could use channels again to send each thread a reference to the stealers Vec, but it seems like there ought to be a better way to do that.

It feels like I'm doing too much channel communication to set up something like this, but I couldn't think of a better way!

Any ideas?

Can you create the Workers and Stealers on the main thread before spawning the child threads, then move a Worker into each child thread along with a reference to the Stealers?

1 Like

I'd love to! Unfortunately, Worker is not Sync, so I get an error if I try to move a reference to it from one thread to another. :sweat_smile:

Actually, now that you mention it, I suppose changing the thread to be defined using a move closure and only taking the reference afterwards could fix that. :thinking: I'll try that tonight!

You could take a look at Rayon's custom tasks. Rayon uses crossbeam-deque under the hood

1 Like

@mbrubeck That totally worked once I switched to a move closure, and it's much nicer than the approach I had before - thanks so much for suggesting it!

Also thanks @Cocalus for the Rayon link. I didn't know about Rayon's custom tasks, and will check out their approach.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.