What's throwing me off with this issue is that while the application is running and connections are timing out, I can visit the sites that it is saying are failing and they work fine in a browser. But I don't know if that's because the OS is allowing another application to establish a connection through some kind of balancing mechanism. I have tried a new Hyper Client instance for each request and that too fails.
I tried it with two applications and it was fine. I think I might have wrongly read in to the maximum amount of connections that my PC can have at one time. I think that's globally and there's still a maximum number of connections/files that a process can have open at one time. Which for my computer is 1024. So I think I need to find a better way of managing the maximum number of connections that are opened at one time and to possibly detect when this is breached and apply some kind of back pressure to the application.
let mut futs = FuturesUnordered::new();
let max = 50;
while let Some(req) = rx.recv().await {
while futs.len() >= max {
// or if you want to store the output, do so here
let _ = futs.next().await.expect("Panic in spawned task.");
}
let fut = HostResolver::do_request(req, client.clone(), request_timeout);
futs.push(tokio::spawn(fut));
}
while let Some(res) = futs.next().await {
res.expect("Panic in spawned task.")
}
I've added that in and its most definitely helped to limit the number of futures. I'm still ending up with more open files that I'd like but it's under the maximum per process as I've checked using:
$ ls /proc/8005/fd/ | wc -l
932
And I've set a maximum number of futures to be 900. So it looks like there's some investigating to do. This is also creating a new client per request.
Tried using the reqwest crate and applying a connect timeout so the timeout is only apply for the connection phase and removing my Tokio timeout.
Creating a new client every X requests.
Creating a new client for every request.
Monitoring the active number of open files. This stays well under the OS limit.
But no joy on anything. The application seems to reach a point and then nearly every request times out and fails. Some do go through and I'm guessing those are ones that were scheduled. The errors returned from reqwest are:
error sending request for url (URL): operation timed out
I'm running this locally. I've wondered this too which is why I've tried accessing the sites that the program says are timing out (which will work fine) and they're fine.
You could try spinning up a cloud vm somewhere and trying it there. I’m not sure what sort of limitations are put in place for this kind of thing, but it’s at least another data point
It appears to work on one of my servers, However, it isn't the fastest so I'm not sure if it's possibly because of it being quite slow.
I would have thought that if an ISP was going to perform blocking then it would block the IP that the requests were coming from. Not just the port? If that's possible.