Hello rust community!
I am currently just playing a bit with Rust and try to do little helper with it. Today I needed a slippy map tile downloader. Basically this is just a HTTP downloader for a lot (think 100k) of generated URLs. Most tile servers have several backends so tiles can be fetched from a list of different hosts. What I did in async Python (no offense, Python is just still my natural language for small helper scripts) was to create on async task per backend host and make all those tasks watch a shared queue. My main task would just put tile paths on those queue and whichever task is free would pick it up and download from its assigned tile server host.
Unfortunately, Python showed some “interesting” behaviour and this made me try to rewrite the small program in Rust for fun. Getting along with futures is pretty easy, they seem quite similar to Java,JS, Python, but with added ownership safety. The point where I had to stop to my surprise is a the queue. After an hour of searching I did not find something equivalent to Python’s async.Queue or Go’s channels that work with tokio. What I have come along and looked most supported was basically
- tokio::sync::mpsc, but that supports only one consumer, but multiple senders. The exact opposite than what I need for my approach
- crossbeam_channel. Functionality looks great, but it seems to do thread primitives only and would probably not play nice with tokio (like blocking a thread instead of giving control back to the worker/event loop)
I found more crates, but those two sounded the most promising.
There is an open crossbeam issue with planned Future support. I guess this would be exactly what I want, but that seems not to be available yet.
Am I missing some queue implementation that fits my needs?
What other approaches to people use to distribute work to multiple tasks?
PS: I think I really want some work distribution mechanism for my code since I would not want to start a new task for every tile that I want to download.