Reqwest slower than Go in GKE Autopilot cluster

help
1

I'm having trouble troubleshooting slow performance with reqwest when running in a Google Kubernetes Engine Autopilot cluster. The exact same code seems to run fine everywhere (even on a Google Compute Engine instance) but the second it is in a GKE Autopilot cluster it is about half as fast as Go.

I'm only testing using http1.1 - I've disabled h2 in my tests.

Here is what I've been testing with: GitHub - bennetthardwick/http-download-testing.

I've put them up as a container - you can test them out like this:

docker run -it bennetthardwick/http-download-testing:ubuntu-24.04

export TEST_URL='some url'

make bench_go # run go | pv > /dev/null
make bench_curl # run curl | pv > /dev/null 
make bench_reqwest # run reqwest | pv > /dev/null

This is the abridged Go code that I've written:

client := http.Client {
	Transport: &http.Transport {
		TLSNextProto: make(map[string]func(authority string, c *tls.Conn) http.RoundTripper),
	}
}

res, err := http.Get(".. url ..")

io.Copy(os.Stdout, res.Body)

This is the abridged Rust/reqwest version that I've written:

let mut stdout = std::io::stdout().lock();
let client = request::ClientBuilder::new().http1_only().build().unwrap();

let mut bytes = client.get(".. url ..").send().await.unwrap().bytes_stream();

while let Some(next) = bytes.next().await {
	stdout.write_all(&next.unwrap()).unwrap();
}

Running in us-central1 everything is between 200-300MiB/s. But reqwest drops down to half when running in GKE.

Here are the results I see:

GCP instance, g1-small, GCS object same region, https

Lang Speed
Go 230MiB/s
Reqwest 210MiB/s
Reqwest (Rustls) 220MiB/s
Isahc 200MiB/s
Ureq 250MiB/s
Curl 260MiB/s

GCP GKE Autopilot Pod, n1 machine class, GCS object same region, https

Lang Speed
Go 240MiB/s
Reqwest 104MiB/s
Reqwest (Rustls) 108MiB/s
Isahc 140MiB/s
Ureq 230MiB/s
Curl 220MiB/s

One thing to note is Ureq doesn't seem to suffer from the slowdown - which starts to feel like it's some kind of tokio related issue. I've tried changing a bunch of settings and nothing helps. If anyone has any ideas I'd really appreciate it!

2

It's somewhat common knowledge that HTTP clients should be singletons so that resources (such as the connection pool) are reused.

ureq does this with the Agent type, which is what ureq::get ultimately uses:

Agents keep state between requests.

By default, no state, such as cookies, is kept between requests. But by creating an agent as entry point for the request, we can keep a state.

Agent uses an inner Arc, so cloning an Agent results in an instance that shares the same underlying connection pool and other state.

The problem with your benchmarking is that you are recreating the client on every request in the case of Reqwest.

3

Hi @firebits.io,

I'm opening a connection to a large file and dumping the contents to stdout - then measuring how fast the download is using the pv tool. There is only a single request in my tests - so I'm not recreating a client every request.

To be clear I'm only seeing the slowdown when running in a pod on a Google Kubernetes Autopilot cluster. When running on a normal GCE instance I don't see any slowdown. I don't understand the differences between these two environments to intuit what could be causing this - but since ureq doesn't suffer from the slowdown it feels like whatever reqwest/hyper is doing compared to ureq must have some part to play.

4

Ah, I see. I inferred that you were running multiple requests. It's certainly puzzling that the observed behaviour only occurs in GKE clusters.

1 Like