What is the best option to send millions of requests

I'm working for a client that needs a batch to empty an old database.

They want to use those data and send millions of REST requests. So I want to introduce them to rust as I love using it for all my projects.

Do you have any hints to give me? I thought about tokio, rayon etc.. but I imagine there is a better crate to send million of requests with the best possible performance. Or there might be a way to do it with tokio or/and rayon

Thanks All :smiley:

Well, tokio and rayon are two completely different things. The former is an asynchronous runtime, particularly useful for fast, asynchronous IO. The latter is a thread pool implementation you can use to parallelize computationally expensive algorithms across threads. What you are probably looking for is a HTTP client library built on top of tokio (or async-std, another asynchronous runtime) for making your requests. Reqwest is well liked and easy to use IMO.

Your DB is probably going to be your bottleneck, whatever you do.

That said, I'd probably use sqlx for the DB, use its row streaming to get the rows one by one, then spawn a tokio task for each, and in the callback, create a request and send it there with reqwest.

To further optimize the process, you can prepare the request builder before the query, and copy it into the spawned tokio task, adjusting only the part of the payload dependent on the DB.

You also need to put some reasonable limit on the number of requests you send to your server. Or to be exact, you should not be fetching further rows from your DB while you have more than that number of requests still running. If you don't put such a limitation, either your rust running computer will go out of memory as it spawns requests for all of your numerous rows, or the HTTP server you're sending the requests to will get overwhelmed, and start erroring on the later requests... So you definitely want this in place.

Here's another topic where a solution to this rate limiting is shown: Tokio: number of concurrent tasks

I should note that I am new to Rust, so I haven't actually used any of this... I'm not sure how you'd adapt the code in the above to the flow I just described.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.