TCP, kernel bypass and the latency tiers

What does TCP do under load, and what does each tier of the network cost per round trip?

TCP: slow start, congestion control, incast

Slow start

A new connection knows nothing about available capacity, and it cannot ask. So slow start begins with a small congestion window (cwnd: how much unacknowledged data it may have in flight) of about 10 segments ≈ 14.6 KB, and doubles it every RTT.

Example: from 14.6 KB to 125 MB takes log₂(125,000,000 / 14,600) ≈ 13–14 doublings ≈ 13–14 RTTs ≈ 1.4 seconds at 100 ms RTT. Short transfers never leave slow start, so their effective speed depends on RTT and transfer size, not link speed. This is why connection reuse and HTTP/2 multiplexing matter much more than they seem to.

Loss-based control must overshoot

Reno / CUBIC (the classic algorithms) use packet loss as the capacity signal. The surprising result: the algorithm must overshoot to learn. It fills the router's queue until it overflows, then slows down. So it normally runs with full buffers, which causes bufferbloat: 100 ms to 1 s+ of extra latency from queueing alone, on a link with no packet loss. Larger buffers made this worse, not better.

BBR (Google's algorithm) estimates the bottleneck bandwidth and the minimum RTT directly, sends at that rate, and does not fill queues. It is much better on long fat networks and lossy links. It is not free: BBRv1 was aggressive towards CUBIC flows. It marks the shift from loss as signal to model as signal.

Incast: the failure that matters most

Incast has exactly the shape of every distributed query: N workers reply at the same time to one coordinator (a synchronized many-to-one fan-in). This is Meera's problem.

1. The switch port facing the coordinator has a small buffer (often a few hundred KB). N simultaneous replies overflow it and packets are dropped.
2. Normally TCP recovers with fast retransmit (the receiver sees a gap because later packets still arrive). But replies are small, so there are no later packets to reveal the gap.
3. So senders wait for the RTO (retransmission timeout), historically 200 ms – 1 s, when the datacenter RTT is ~100 µs.

One lost packet costs a 1,000× delay. Throughput does not fall slowly; it collapses by about 10× once the number of senders passes a threshold.

Mitigations:

• microsecond-level RTO timers;
• ECN / DCTCP (switches mark packets before dropping them, and senders react in proportion);
• staggered / jittered requests (small random delays, so replies do not arrive together);
• limit the fan-out width.

Jitter works by destroying synchronization — the same idea as the variance-reduction methods in the CPU part.

---

Kernel bypass

• Same arithmetic as the storage finding. The kernel network path costs ~5–15 µs: thousands of cycles of interrupt, context switch, copies and cache pollution. When the wire RTT was 500 µs, that was noise. At 100 Gb/s and ~2 µs RDMA or kernel-bypass RTT, software is the main cost. Cause 3 moved up again.
• DPDK (Data Plane Development Kit): user-space drivers that poll the card, with no interrupts, huge pages and pinned cores. ~5 µs → under 1 µs, with near line-rate small-packet processing. The price is real and often decisive: one core spins at 100% forever, you lose the whole kernel stack (TCP, firewall, tools), and the card is dedicated.
• RDMA (Remote Direct Memory Access) is more interesting: it reads and writes the memory of another machine directly, without using that machine's CPU. ~1–2 µs end to end. Remote memory then costs about the same as a remote NUMA access in CPU and memory, which made disaggregated memory and CXL-over-network realistic.
• Rule for when it matters: only when the work per message is similar to the ~5–15 µs stack cost. Analytics moving 8 MB Parquet chunks? Irrelevant — you are bandwidth-bound. Distributed transactions, consensus, shuffles of tiny messages? Decisive. For most teams the middle path is io_uring + TSO/GRO + interrupt tuning, which gives much of the benefit and keeps the kernel.

---

The latency tiers

This table is the network's "numbers every programmer should know". You will use it often.

• Each tier is about 10× the previous one, a useful mental ruler: same host ~10 µs, same rack ~100 µs, same AZ ~500 µs, cross-AZ ~1 ms, cross-region ~100 ms.
• The last row deserves attention. Cross-AZ RTT is ~1 ms, but an S3 GET in the same region takes 20–100 ms. The extra 20–100× is not distance. It is load balancers, TLS handshake, authentication, metadata lookup and internal replication. The minimum is service depth, not travel time. Two different mechanisms with the same units. Mixing them up leads people to "fix" S3 latency by moving servers closer, which does not help.
• Synchronous replication across AZs costs ~1–2 ms per round trip. This is the numeric basis of the multi-AZ trade-off, and why quorum systems (which need a majority of replicas to agree) keep replicas inside one region and replicate asynchronously between regions.
• Synchronous consensus across regions is impossible for interactive work. Mumbai ↔ Virginia is ~200 ms RTT, and Paxos/Raft (consensus protocols) need 1–2 round trips per commit. This is geometry, not tuning. It is why Google Spanner uses atomic clocks: TrueTime lets it skip one coordination round trip, and pay a commit-wait limited by clock uncertainty rather than by distance.

---

The shape of the communication

The shape of the communication matters more than its volume. All-to-all shuffle is costly because it breaks the oversubscription bet. Synchronized fan-in causes incast because it is synchronized. Chatty chains are slow because RTT cannot be amortized. The same bytes can cost very different amounts depending only on the pattern.