Some updates on the above highlighted open questions:
Does hyper-tls or native-tls or tokio-tls offload packet decryption from the reactor thread?
Nope. According to @sfackler (on #tokio-users discord, thanks!) there is no provision for offloading TLS en-/decrypt from the reactor threads (at least with the default TLS stack used here). However TLS specifies a maximum record size of 16KiB, so "you won't be doing more encrypt/decrypt work than that per call" and "hardware-accellerated AES runs at multiple gigabytes per second iirc". So this is a simpler case and (in my testing) unlikely to benefit from any attempt to move off a reactor thread.
However (and thanks again @Nemo157) with additional testing I do find that I'm receiving
Bytes items from hyper 0.13.1's response body
Stream at upwards of 512KiB in size, by default. There is an easy change for comparative benchmarking:
let client = hyper::client::Client::builder()
+ .http1_max_buf_size(64 * 1024)
And this has measurable effects on the client benchmarks described in my original blog post:
On my laptop (less important) with in-process server, it actually makes fs and mmap (tmpfs) benches significantly faster.
On the ec2 virtual hosts (more important!), it makes things considerably slower (particularly the ram benches).
For approximately size-linear operations like decompression, I find that significant. Beyond the hyper client config tweak, my
Sink implementations are in position to potentially do partial (sub-chunked?) processing. Hopefully I'll be able to compose that with another
Sink wrapper type over my 6 others. Splitting the original
Bytes or (1)
MemMapBuf into many via something like
Bytes::split_to and more
As for my thesis, I'll be moving forward with using synchronous (blocking) tmpfs I/O on reactor threads in production, with the library retaining the option to offload (async) these, which thus far is really just a complicating solution in search of problem.