Which network choices change behavior

Which network mechanisms change what your system means, and which only change how fast and how cheaply bytes move?

CaseConstructed case · Lena at Cartwheel

A high bill, morning spikes, connections that hang

Lena's pipeline now sends millions of small events, each about 200 bytes, from the Orders service to Kafka and on to the analytics warehouse. After a few weeks she sees four things that look like bugs.

The network bill is high. Cartwheel sends data between two cloud regions, and the provider charges for every gigabyte that leaves a region. The events are small, but there are very many of them.
CPU is spent in the wrong places. The profiler shows that a large part of the producer's CPU goes to compression and to TLS (the encryption layer), not to Cartwheel's own code.
The p99 latency spikes after quiet periods. At 3 a.m. almost nothing happens. At 6 a.m. the first requests of the morning fail or take seconds, and then everything is normal again.
After a load balancer reboot, some services hang. The service shows as connected, but no data arrives. Sana (on call) calls these connections "half-open": one side thinks the connection is alive, and the other side has forgotten it.

Nothing here is a logic bug in Cartwheel's code. Each symptom is a property of how bytes move over a network. This article explains the mechanisms behind them. At the end we go back to Lena and fix all four.

How it ends →

A frame, and an honest sort

The articles on transport, flow control and routing were about semantic decisions: choices that change what your system means. A queue and a log give different answers to "can I replay?" A partition key decides which order exists. A dead-letter queue trades completeness for availability.

Most of this article is different. Zero-copy, connection pooling, keep-alive and TLS termination are mainly constant-factor optimizations and operational hygiene. They change how fast and how cheaply records move. They do not change what the movement means. For these you should know that they exist, know the one or two failure modes of each, and look up the details when you need them.

Two items are exceptions and deserve real attention:

Protocol framing decides where a malformed record fails: at write time, in the producer, or at read time, in the consumer. That decides who has to debug it.
Compression interacts with batching, partitioning and CPU budgets in ways that regularly surprise people.

So these two get depth, and the rest get what they are worth.

One idea organizes everything below. Each mechanism answers one of three questions:

Question	Mechanisms
How do I avoid paying a fixed cost again and again?	connection pooling, keep-alive, zero-copy, coalescing (Nagle)
How does the receiver know where a message starts and ends?	framing
What am I willing to spend to move fewer bytes, or safer bytes?	compression, TLS

(A fixed cost is a cost paid once per operation whatever its size, such as the setup of a connection. Paying it once and sharing it between many records is the idea behind batching, which Flow control explained.)