I wrote a terse HTTP webserver using tokio. It's hard for it be simpler (it cheats and doesn't pay attention to request headers; and it doesn't send a Date header back ). And yet, actix-web is outperforming it by a decent margin in benchmarks. Here's the whole code:
https://gist.github.com/fpgaminer/d2e459e247bfbd2fd7ade42ed466e0e8
The actix benchmark code from TechEmpower's benchmark gets 50k req/s
in my tests; my code gets 43k req/s
. This is on cloud instances, 4096 keep alive connections, wrk
, 2 vCPUs per machine.
Can anyone think of where this difference could come from?
Because of the use of keep-alive connections I think that none of the performance difference is due to the way the listener is setup. So the fact that actix-web uses a different backlog setting and possibly multiple listeners doesn't seem to matter (though I did try a few modifications of my code to test and didn't see much difference). So it must be something else. actix appears to mostly use mio directly rather than tokio's runtime and reactor. Does using tokio have that much overhead?
Here's the full text of the benchmark results:
## actix, no pipeline:
Running 15s test @ http://10.142.0.3:8080/plaintext
2 threads and 4096 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 10.09ms 505.14us 23.99ms 70.69%
Req/Sec 50.47k 2.10k 55.84k 65.33%
Latency Distribution
50% 10.15ms
75% 10.46ms
90% 10.63ms
99% 11.09ms
1506079 requests in 15.05s, 186.72MB read
Socket errors: connect 3077, read 0, write 0, timeout 0
Requests/sec: 100059.44
Transfer/sec: 12.41MB
## tokio-raw, no pipeline:
Running 15s test @ http://10.142.0.3:8080/plaintext
2 threads and 4096 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 11.52ms 7.57ms 237.87ms 63.74%
Req/Sec 43.86k 3.55k 49.92k 62.67%
Latency Distribution
50% 11.40ms
75% 17.27ms
90% 21.29ms
99% 23.96ms
1309184 requests in 15.05s, 97.39MB read
Socket errors: connect 3077, read 0, write 0, timeout 0
Requests/sec: 86962.04
Transfer/sec: 6.47MB
You can also see that the latency distribution for actix is flat, while there are a number of slower requests for my code.
I've tested tokio-minihttp, hyper, and the HTTP example in the tokio repo. All perform even worse than my code, let alone actix.
TechEmpower's benchmark confirms this: TechEmpower Framework Benchmarks You can see that on Cloud hardware actix-raw outperforms tokio-minihttp. (NOTE: This is not the case on Physical hardware benchmarks, but I suspect that this is due to the hardware used in Physical tests being more powerful, thus the benchmarks hit the network bottleneck and so you can't see their difference in performance anymore).
In practical terms, I'm not concerned about performance at these levels on artificial benchmarks. But I am curious as to why my exceedingly simple code is being outperformed by code that is doing an order of magnitude more work.
Thank you!