I found a general answer how to make concurrent requests using the crate reqwest on StackOverflow here.
I also searched this forum, but did not find the answer I am looking for, so here goes:
To get the information I want, I have to make GET requests to an (in theory) unknown number of pages, which means in each iteration of the loop something like ?page=1
or ?page=2
gets appended to the URL.
When the API has no more information for me, it returns a String with []
(empty brackets) as content, which currently happens for me if the page_num
is 11, but of course for other users this will be different.
Now since each "page request" takes 1-2 seconds, I want query the individual "page urls" concurrently, and when one of the "page requests" returns []
, break out of the loop.
for page_num in 1..=20 {
// request_a_page builds the page specific URL and a Reqwest client,
// then makes a GET request and returns Result<String, Box<dyn Error>>
let body = request_a_page(page_num, token_info)
.await
.unwrap_or_else(|_| panic!("failed to get page {page_num}"))
// currently true for if page_num == 11
if body == "[]" {
log::info!("empty response --> break");
break;
}
// ... process body ...
}
I have previously written a Go version of this, where I simply started multiple goroutines with a sync.Waitgroup. At one point one of the goroutines would set a flag (struct with sync.Mutex) value, so no more new goroutines would be started (as a workaround/temporary fix I added a time.Sleep(250 ms)
after page_num is higher than the number of pages that I know I currently need to query, so that Go does not start hundreds of goroutines in the meantime).
Since I am new to Rust and mostly code as a hobby, I am currently not able to translate this (even with help from the Stackoverflow example above) into Rust code, so any help on this would be appreciated.