Reqwest: Multiple concurrent requests by priority

help
1

Suppose I want to send 1000 requests using reqwest to a server supporting HTTP2. The server indicates that it supports 128 concurrent streams. I would like requests earlier in the list to be executed before later requests. I would also like to use the maximum concurrency possible.

I could spawn 1000 tokio tasks, one for each request, but that there is no guarantee on the order in which the requests are sent. And many of those tasks would be waiting for other requests to finish. It is also inefficient to be spawning so many tasks constantly.

I could also spawn 128 workers tasks, one for each concurrent stream supported, and send the requests to the tasks over a priority queue, with the requests being taken from the queue when ready and then sent to the reqwest client.

This raises the question of whether reqwest really sends 128 concurrent requests, and also whether reqwest might be able to open an additional connection if there are more requests than possible concurrent streams. The number of workers would need to be changed if the number of concurrent streams changes.

What is the best way using reqwest of maximising concurrently executing requests while also respecting a priority of the requests? Are there any better alternatives to reqwest for this sort of thing?

2

This seems like a great way to achieve it: Reddit - The heart of the internet

3

This does not answer the question of how to send requests by priority while maximising concurrency.

4

Good point, but I would expect this to hold if the vec of urls is sorted.

5

The requests will be coming in over a priority queue which will be updated dynamically. Also the link does not answer how to maximise concurrent tasks per the number of concurrent requests that reqwest can perform while preserving priority

6

The amount of concurrency is demonstrated in the provided example.

stream::iter(0..1_000_000)
        .for_each_concurrent(10, |url| callback(&client, url)).await;

In this case up to 10 requests will be run at once. Just change that number to 128. For your original scenario of "send 1000 requests to the server" this should be sufficient.

If instead you have a priority queue continuously providing values, then something like this could work:

async fn process_requests() {
    let st = stream::iter(0..)
        .then(async { client.get(fetch_url_from_queue().await).send().await })
        .buffer_unordered(128);

    while let Some(val) = st.next().await {
        // do something with the returned http request
    }
}

Note that you probably should use some kind of rate limiter, like governor

7

Thanks. I actually already have a custom rate limiter that I will be using.

What I am more trying to get to the bottom of is how to have a number of workers equal to the maximum number of concurrent requests that can be sent with reqwest over a single client.

It is unclear to me whether reqwest is going to max out the number of concurrent streams, and whether it also might create a new connection when the number of concurrent requests exceeds the HTTP2 maximum number of concurrent streams.

It seems that reqwest doesn't really expose how many streams/requests it is sending at once, and there doesn't seem to be a way of coordinating this.

Can you advise on how better to coordinate with reqwest on the number of concurrent requests it is executing?

8

Reqwest is a higher level API for http requests. If you need more specific control you could use hyper, but it is much harder to use. As long as you are using the client struct reqwest should reuse connections where possible. You mentioned you're already using a custom rate limiter? I would look at how many concurrent requests can max out that limit and go from there.

9

Note that while you're executing the body of the while let loop there will be no way to poll the stream, which can e.g. cause timeouts. This can be especially problematic if you .await inside it.