Causes for differences in `TcpStream` connection time

help
1

In both linked playgrounds, a sequence of TCP connections are initiated and the time until connection is completed is stored per iteration. In the first, initializations are done in a tight loop. In the second, there is a delay between each connection.

  1. Tight loop connections
  2. Delays between connections

While timing will vary, I'm seeing roughly the following times.

// First
Connection times: [99.462µs, 29.681µs, 26.671µs, 25.07µs, 26.56µs]
// Second
Connection times: [65.541µs, 85.592µs, 64.481µs, 46.461µs, 63.371µs]

In the first example, I'm observing the first connection (generally) takes noticbly longer than the subsequent connections. Ih the second example, I'm observing the connections take generally comparable time.

In the first example, what could the cause of this noticeable difference in the first connection be?

I tried a strace dump of the system calls made in each, and they seem comparable (minus the additional __semwait_signal calls on Mac for the additional sleep calls in the second example). I tried various delays in the second option (in us, [10e2, 10e3, ..., 10e6]) and prior to 10e6 I notice differences in the first connection time, similar to the first example.

I wondered if the cause was due to TCP "caching" and "cold starts" (similar to what's mentioned in this article) but I think caching should be done per host:port and changing the port number per connection shows the same effect as in the first example (playground).

2

To reduce the amount of ack packets sent, there is something called delayed TCP Delayed Acknowledgment where the initial packets wait for a while in anticipation of more ack packets to ack them together, it could be the reason for the extra delay
Furthermore try disabling Nagles algorithm and run the tests again see if the difference is less this time.

3

Thanks for the ideas. However, I think these optimizations affect packets sent after the connection is established (delayed ACK from receiver, Nagle from sender), but the latency measurements are over the initial TCP handshake.