Calling lots of async functions at once and getting the results

help
#1

Rust noob here, I am building a web app using Actix and it fetches data from a lot of external APIs.
I have a bunch of modules for each API and a same-name async function in each one of them. They are all API calls but to different APIs with different structures (hence multiple modules). In my web app, I want to return a JSON array of these results (I am putting them into the same format/struct) and have a const Vec of API structs as my global state to keep track of the names of the APIs and their supported locales/languages.
What is the best way to achieve this? Am I thinking this all wrong?

#2

You should use the futures crate for this. Let's say you have a vector urls with the information needed to make a request, and an async fn fetch_url that performs one request. Then you can do this:

use futures::stream::StreamExt;

let responses: Vec<_> = futures::stream::iter(urls)
    .map(|url| async move { fetch_url(url).await })
    .buffered(20)
    .collect()
    .await;

or if fetch_url returns a Result:

use futures::stream::{StreamExt, TryStreamExt};

let responses: Vec<_> = futures::stream::iter(urls)
    .map(|url| async move { fetch_url(url).await })
    .buffered(20)
    .try_collect()
    .await?;

The 20 is the max number of connections at the time.

#3

Thanks for the reply. However, the problem is that the functions can't be the same. The APIs have different HTTP methods and different formats and parsing logic. Any way to handle that?

#4

Make a function with a big if or match for each case?

1 Like
#5

Hoped for something more elegant but thanks!