I'm a big fan of the futures crate's futures::stream::buffered for making a bunch of requests (or completing other long time tasks) but at most n at a time. It got me thinking if it's possible to do the same in non-async Rust.
I can recreate a somewhat similar thing using a mpmc channel:
But it takes quite a number of lines so I would like to make a helper out of it. I'm unsure how to type the inputs and the task's closure. The parameters do not live long enough and I cannot use 'static as I want to use for example a single reqwest Client for all requests.
Anyone have ideas how to move forward? Or perhaps a library that already exists for this.
The moment you pass your types to thread::spawn, they need to have the + 'static lifetime bound present. But you already have your answer right there in your code: Arc. You wrap your closure in an Arc and cheaply provide a clone to each thread instead of a reference. You could do the same for your reqwest Client, when you move it into your closure. You don't even need to wrap it in an Arc, as Client wraps its internal connection pool in an Arc, so you can cheaply clone it while keeping the same connection pool: client.rs - source.
Another unrelated topic here on URLO gave me an idea how to make it work without the + 'static requirement, scoped threads. They don't require + 'static and the helper function is simplified along the way:
Or perhaps a library that already exists for this.