Zero-copy, Nagle and TLS

Which constant-factor mechanisms do you benefit from by not defeating them, and how do layered optimizations combine badly?

Zero-copy

The normal path for sending a file over a socket is: disk, then the kernel's page cache (the operating system's memory cache of file data), then a user buffer (your program's memory), then a kernel socket buffer, then the network card (NIC). Two of those steps copy the data through user space, and the path needs 2–4 context switches (moments when the CPU switches between your program and the kernel). All of this moves bytes that your application never looks at.

sendfile() and splice() skip the trip through user space. The data goes from the page cache to the socket buffer, or, with scatter-gather DMA (the card reads from several memory locations by itself), the network card reads directly from the page cache. It is one system call and no copies by the CPU.

The precondition is the key point: your application must not need to touch the bytes. Zero-copy works exactly when you are only a conduit. Kafka's fetch path is the standard example. The broker serves stored bytes to consumers without interpreting them, so it can call sendfile() straight from the log segment file. This is a significant part of Kafka's throughput.

It is also why the compression point above matters so much. The moment the broker has to decompress, transform or re-timestamp a record, zero-copy is lost, and you are back to copies plus CPU work. The same is true for software TLS, because encryption must read the bytes. (Kernel TLS, kTLS, gets zero-copy back by moving encryption into the kernel. Hardware offload does the same inside the network card.)

RDMA (Remote Direct Memory Access) is the extreme case: the remote network card writes directly into the target machine's memory, bypassing the remote CPU and kernel entirely. Latency is below a microsecond. It needs specialized hardware and a lossless network fabric, and it changes the programming model, because there is no "receive" event by default. It matters in high-performance computing and some high-end storage. You will not choose it in a normal data platform. It is useful to know as the floor of what is physically achievable, so that you can judge how much of your latency is protocol overhead and how much is physics.

Practical takeaway: you rarely implement zero-copy. You benefit from it by not defeating it. Do not put a transformation into a path whose value is being a conduit. If you proxy a stream and find yourself deserializing every message just to add a header, you have probably cost more than the header was worth.

Nagle-style coalescing versus low-latency flush

This is the batching trade from Batching, one level lower.

Nagle's algorithm: do not send a small segment while an earlier small segment is still unacknowledged. Under load, small writes pile up and leave as full segments. When the connection is idle, data goes out at once. This is self-tuning batching that uses the receiver's own state (the missing ack) as its signal. It is the kind of adaptive pattern Flow control praised.

The problem is delayed ACK. A receiver may hold its acknowledgment for up to about 40 ms, hoping to attach it to data it is about to send anyway. Now put the two together:

Nagle on the sender waits for the ack before sending the next small piece.
Delayed ACK on the receiver waits for data before sending the ack.

Each side waits for the other until a timer fires. This is the classic 40 ms cliff for request-response traffic with small writes. A call that should take 1 ms takes 41 ms.

Each side waits for the otherillustrative · 1 ms RTT · 40 ms timer

TCP_NODELAY turns Nagle off. In practice: enable it for interactive and RPC traffic (nearly all modern libraries already do), and do your batching explicitly in the application, where you can see it, measure it and tune it against a real latency target (an SLO).

import socket
s = socket.create_connection(("inventory.internal", 9000))
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)   # send small writes immediately

The lesson that generalizes, and the reason this item is here at all: two independent latency-hiding optimizations at different layers can combine into a latency disaster. Nagle is reasonable. Delayed ACK is reasonable. Together they stall. Whenever you add buffering at one layer, ask what the layer below already does. The failure will not look like either mechanism. It will look like an unexplained stall of fixed length.

TLS: what it costs a crossing

TLS encrypts a connection and lets the client check the server's identity. What it protects, where your trust boundary sits when you terminate it at a load balancer, and why mutual TLS gives each service a cryptographic identity, are in Encryption in transit and Authentication. For moving records, three points matter:

The handshake is the cost, not the encryption. Bulk encryption with AES-NI is single-digit percentages of CPU; the handshake is asymmetric cryptography and 1–2 round trips. So session resumption and connection reuse matter a great deal, and a pool that churns connections costs much more under TLS than without it.
TLS 1.3 cut the handshake to 1 RTT and offers 0-RTT resumption, but 0-RTT data can be replayed by an attacker, so use it only for idempotent requests.
The remaining real cost is the loss of zero-copy, because encryption must read the bytes, unless you have kTLS or network-card offload. For a broker that serves large volumes of stored bytes, that can matter more than the cryptography itself.

"TLS is too slow" is, today, almost always false for bulk traffic and almost always true for connection-per-request patterns. Fix the connection pattern, not the encryption.

What actually matters here

Compressed to the essentials:

Framing is the one semantic choice. Length-prefixed means early binding: failures show up at write time, in the producer. Delimited means late binding: failures show up at read time, in someone else's job. Text formats survive because of splittability. Binary formats solve splittability with block framing plus sync markers, which is the pattern to know. Always enforce a maximum frame size.
Compression is coupled to batching and partitioning through the batch size, and the coupling is invisible in configuration. Zstd at a low level is the modern default. Compression should be end to end: every intermediary that touches it pays a full CPU cycle and kills zero-copy.
Pool exhaustion policy and heartbeat timeout are two settings you probably inherited as defaults, and both are load-shedding decisions. Block-forever pools and a single timeout for both liveness and progress cause a disproportionate share of production surprises.
Layered optimizations compose badly. Nagle plus delayed ACK is the standard case, but the pattern is general: buffering at two layers, framing disagreement between two hops, compression at two levels. When you add a mechanism at one layer, ask what the layer below already does.
The rest (zero-copy, keep-alive, TLS) you benefit from by not defeating. Do not put transformations into conduit paths. Do not churn connections. Do not terminate TLS without knowing you have declared a trust boundary.

Case, resolvedConstructed case · Lena at Cartwheel

A high bill, morning spikes, connections that hang

Small 200-byte events ran up a cross-region egress bill, producer CPU went to compression and TLS, the first requests after quiet nights failed, and some connections hung half-open after a load balancer reboot. Where it started: Network-level mechanisms →

Lena goes through her four symptoms with the mechanisms of this article.

The high bill. Her producer had been lowering linger.ms to keep latency low, so batches held only 3–5 events. Per-event compression ratios were about 1.2×. She raises linger.ms from 0 to 20 ms, which at Cartwheel's rate fills batches of a few dozen events, switches the codec to Zstd level 3 and turns on compression only on the cross-region link. The ratio rises to about 6×. Egress falls to about one fifth. (Illustrative numbers.) She also checks that the broker does not recompress: the codec on the topic is set to "producer", so the broker forwards batches as they are.

CPU spent on compression and TLS. Inside the datacenter the links are fast, so the slow gzip codec she had used was the bottleneck. She uses LZ4 for local hops and Zstd only on the expensive link. For TLS, the CPU cost was not the encryption. It was handshakes, because a client library opened a new connection for each batch. She switches it to a pooled connection with one pool per downstream, sized by Little's Law (throughput × latency), and the handshake cost disappears.

Spikes after quiet periods. At night the connections are idle. The load balancer's idle timeout (350 seconds) is shorter than the pool's idle time (1 hour), so it removes them silently, and the first morning requests use dead connections. Lena sets the maximum idle time to 60 seconds, below the load balancer's timeout, enables application-level keep-alive pings and turns on validate-on-borrow for the few services that cannot repeat requests safely.

Half-open connections after the load balancer reboot. The reboot dropped the connection state without sending a reset, so neither endpoint was told. With TCP keepalive at its 2-hour default, the dead connections would stay "connected" for hours. The application-level heartbeat now detects them in seconds. She also splits the consumers' liveness timeout from their progress timeout, as Kafka does, so a long processing batch is no longer mistaken for a dead consumer.