I'm having a binary, which is basically glue code between reading in data, putting it into templates, and run those templates through latex. Latex is of course an external process, and takes the majority of the time needed. The scenario consists of roughly 15-30 latex invocations, so I'd like to uses parallel processes as much as possible.
What's the way to do that? I thought about using rayon's par_iter or something that gives me workstealing, but I'm not sure if workstealing interacts nicely with external processes (I guess it does), and I'm wondering if there's another way to do this nicely. I'm thinking of "just" using spawn to spawn a number of processes and collecting the results afterwards, but I'd kinda have to deal with making sure there's always a certain number running, and I'd have to determine how many to run, maybe dynamically from the number of cores.
You can "just" spawn new thread per each invocations. 30 threads are next to nothing within the human noticable timescale. Normal processors can easily handle thousands of them running concurrently.
Given that your Latex work "takes the majority of the time needed" I would not be inclined to worry about the overheads of spinning up a few processes to run them.
You might like to ensure you don't spin up more Latex processes than there are cores on your machine at the same time. I think it's probably not needed but it would ensure you are not trying to use many times your actual commute power at once and that you don't run out of memory while trying.
To be clear, I'm not worried about any overhead indeed, but I'm worried about 30 latex processes running at the same time on an average laptop. Yeah, I figured I'd need to restrict it to "number of cores" many, but that will need a bit of bookeping on my side, and I was kinda hoping there was an easy way out like par_iter.
The average PC/laptop now a days is running hundreds of processes all the time. Just the OS itself will be running surprisingly many, before we start to think about what applications you want to run. Open up your task manager or whatever and tae a look.
Certainly there are overheads in spinning up processes, greater than threads, but given that most of the time is being spent in Latex doing the heavy work, and you only have a handful of processes, those overheads will be unnoticeable.
I'm not sure we're on the same page. Overheads don't matter, but the processes themselves do pretty heavy work, and I'm worried about maxing out the laptops capacity so the user can't do anything anymore untill they're all finished.
(e) That said, any hint on how to achieve what join does for 30 instead of 2 closures? I just want to stick them in the pool and wait until they're all finished (just like join does...)? Thanks!
Yes, that is what I said in my first reply. You might like to limit the number of processes you start at the same time to the number of cores you have. Perhaps even leaving one in reserve. That will keep the system responsive. And also ensure you don't run out of memory and run into using swap space.
How you achieve that is another matter. You could queue up your work "batches" and have a bunch of threads remove them from the queue and run them.
Or let something like rayon take care of that work scheduling.
You can use a mpsc channel from std. use spawn from rayon to start each task, moving a clone of the mpsc Sender into each task. And then use the send ends of the channel to send the results of each task back to the main thread. And then in the primary thread just recv all the results.