I have a process that downloads and processes tens of thousands of resources via HTTP (using reqwest blocking).
On each run, a very small number (~.1%) of these GET requests fail with an error of this type:
Transient(reqwest::Error { kind: Request, url: "https://api.<x>.com/eap/content/responses/?requestIdentifier=c2011644", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) })
I've seen it suggested that this means the server is closing the connection before the response is streamed. The guys on the other side say no way.
Anyone have any experience getting around this? Is it actually a reqwest bug?
I'd probably get some TCP dumps if I wanted to pursue blame.
However, no network is infallible. So it's probably sensible to build in some sort of recovery mechanism to your application in any case.
So - I am using the backoff crate (retry) for that, and the crazy thing is, when one of them fails, it fails multiple times in a row.
1 Like
x0rw
4
It's probably a rate limiter implemented by that server or similar mechanism
You'd think so, but they have other rate-limiting specs, and correctly return a 429 if you exceed them.
The process does rerun the failures. It's just something I'd like to get resolved.
The whole path is complex: POST a request, GET to monitor progress, then the final GET to get a (massive binary) payload when complete.
x0rw
6
I've never used reqwest but there should be an option to increase the timeout, try it maybe