I'm trying to create a program that runs multiple instances of git concurrently, using the rayon crate and a ThreadPool. The code I've written so far is as follows:
for repo in dir {
pool.install(|| fetch(&repo).unwrap());
}
This code is clearly problematic: it doesn't run concurrently. I thought that using a thread pool, I could have the tasks execute in a way where the thread pool consist of, let's say 4 worker threads, and each worker thread takes in a new task after the previous task completes, continuing this until all tasks have been completed. This sounds like the optimal way of doing it, the question is just: how? Is rayon even the best approach for this or should I use something else? Do I even need a thread pool to execute subprocesses concurrently?
I'm using Tokio / async to do something similar. So far the work is going well. The one hitch-in-my-get-along has been the tokio::select macro. The did-not-complete branches are canceled. That means any state stored on the stack is discarded.
Maybe. It depends. A dozen threads is not troublesome. Thousands of threads are a problem. You have to weigh the system costs against your time to develop a more elegant solution.
One approach is to use the Command::spawn() function to start each subprocess in the background, then wait for them all to complete. The exact code will depend on what you are doing, but it could look something like this:
use std::process::{Command, Child};
let mut children = Vec::new();
// spawn all child processes in the background
for repo in dir {
let child = Command::new("git").args(...).current_dir(repo).spawn()?;
children.push(child);
}
// once the child processes have started, we can wait for them to finish
// and get their output
for child in children {
let output = child.wait_with_output()?;
if !output.status.success() {
...
}
}
That way we can use a single thread for orchestrating everything and don't need to worry about something complex like async/await. The OS will handle all the multitasking for us.