Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

The network: a shared medium with invisible neighbors

Why is a network different from a disk, and why do networks collapse under load?

The medium is shared

In storage and CPU, the medium was passive. A disk platter has no opinions. You made a request and physics answered. The network is different:

This one change explains almost everything special about networks. Cause 1 (distance) sets a minimum no engineering can lower. Cause 3 (fixed cost per operation) returns as per-packet cost. And a new mechanism appears that had no earlier equivalent:

Congestion control is distributed feedback control under uncertainty. Congestion means more traffic than the path can carry. No sender knows the lengths on its path, the number of competing flows, or the free capacity. Each sender guesses from packet loss and delay — signals that arrive one round trip late, after the harm is done. A control system with delayed feedback tends to oscillate.

Remember the CPU idea: the machine is a guessing machine. Here it is the same, one level up: TCP guesses the available , and it can only learn by overshooting. Packet loss is not only a failure; it is the measuring instrument.

Latency and bandwidth are independent

  • They are different physical quantities that improve in different ways. Bandwidth is bought with parallelism: more lanes, more wavelengths of light, better encoding. Latency is limited by distance ÷ signal speed, and you cannot buy a shorter distance. So link bandwidth rose ~1,000× since the 1990s while RTT barely changed.

  • "The network is slow" usually means latency-bound. "The network is saturated" means bandwidth-bound. Different diagnosis, different fix, often confused.

  • The minimum comes from geometry. Light in fiber needs ~5 µs per km (about 20 cm/ns), and real cables are 1.3–1.5× longer than the straight line because they follow existing routes. Example: Mumbai → London is ~7,200 km in a straight line → ~9,000 km of fiber → ~45 ms one way → ~90 ms RTT minimum, forever. Add switches and routers (~1–10 µs each, more when queues form) and protocol processing.

  • The bandwidth-delay product (BDP) connects the two, and it is Little's Law:

    Example: a 10 Gb/s link with 100 ms RTT needs 10 Gb/s × 0.1 s = 1 Gb = 125 MB in flight to stay full: the sender must keep sending that much before the first acknowledgement returns. Default TCP buffers (a few MB) limit you to ~1 Gb/s or less, whatever the link capacity. This is the "long fat network" problem, and the reason one cross-region stream is slow while 20 parallel streams fill the link. It is the same equation as queue depth and cache-miss concurrency.

  • Practical view: latency decides how fast a chain of dependent steps finishes; bandwidth decides how much independent work you can push. A query with 5 sequential round trips needs at least 5 × RTT, whatever the bandwidth. This is why chatty protocols fail over long distances, and why is the universal fix.

Bisection bandwidth and oversubscription

  • Cause 2 applied to network switches. Full any-to-any bandwidth needs switch capacity proportional to the number of ports, which is expensive. So designs amortize. Example: a rack has 40 servers × 25 Gb/s = 1 Tb/s of server bandwidth, but only 200 Gb/s of uplink out of the rack. That is 5:1 oversubscription, a bet that most traffic stays inside the rack.
  • Bisection bandwidth = the worst-case across a cut that splits the network into two halves. It is the honest capacity number, and it decides performance (a shuffle is the all-to-all data exchange in distributed query engines like Spark). Marketing quotes port speed. Shuffles use bisection.
  • Modern Clos / leaf-spine networks can be built non-blocking (1:1), and cloud providers increasingly do so. But the locality assumption is still built into the software on top: rack-aware replica placement in HDFS, Spark locality levels, zone-aware scheduling. All exist because the network is cheaper when you stay local.
  • The key consequence: shuffle is the one operation that deliberately breaks the locality assumption. All-to-all traffic is exactly the pattern the network was priced to avoid. This, not disk and not serialization, is the main reason shuffle is expensive.
  • ECMP (Equal-Cost Multi-Path) chooses a path by hashing flows, not packets, so a few huge flows ("elephant flows") can collide on one path while other paths are idle. is a limit you often cannot reach.

MTU and per-packet cost

  • Fixed cost per operation, once more. Every packet pays for headers (~54 B for Ethernet + IP + TCP) and for processing at both ends: an interrupt, a DMA descriptor, a checksum, protocol state, waking the socket. The cost is per packet and nearly independent of packet size.
  • This gives the natural size, as in Storage media. Example: with the standard (maximum transmission unit, the largest packet) of 1,500 B, a 10 Gb/s link carries ~820,000 packets per second, leaving ~1.2 µs of CPU time per packet — about the same as the processing cost. At 100 Gb/s it is ~8.2 million packets per second and ~120 ns per packet: a general-purpose network stack cannot handle that per packet. Hence offloads (hardware does the work).
  • Jumbo frames (9,000 B) cut the packet rate 6×, raise header efficiency from ~96.5% to ~99.4%, and, more importantly, cut per-packet CPU 6×. Condition: every device on the path must agree. PMTU black holes (Path-MTU discovery uses ICMP messages that firewalls often drop) cause the worst failure in networking: connections that start normally, then hang at the first large transfer.
  • TSO / GRO / LRO (TCP Segmentation Offload, Generic/Large Receive Offload) are the software answer: the kernel gives the network card a 64 KB buffer and the card splits it into packets. You get jumbo-frame CPU cost on a normal network. It is also why tcpdump shows 64 KB "packets" that never existed on the wire.