Nagle plus delayed ACK: reproduce the 40 ms cliff
Measure your machineNeeds Python (Linux shows the effect)
Problem
Two TCP optimisations, each sensible alone, interact badly. Nagle's algorithm on the sender holds back a small write while earlier data is unacknowledged, hoping to combine it with more. Delayed ACK on the receiver holds back its acknowledgement for up to about 40 ms, hoping to piggyback it on a reply. A client that sends a request as two small writes (a header, then a body) and waits for the reply triggers both: the body waits for the header's ACK, and the ACK waits for a reply that cannot come until the body arrives.
Write a client that sends a small request in two separate send() calls (a header of a few bytes and then a body) and waits for a short reply. Run it against a simple echo-style server on localhost or, better, on another machine. Measure the round-trip time with Nagle enabled (default). Then set TCP_NODELAY on the client socket and measure again. Also try sending header and body in one send() call: it removes the stall without any socket option.
The reference run is on Linux, inside a container: macOS's loopback does not show the stall.
The question. How long does a two-write request take with Nagle on, with TCP_NODELAY, and with header and body in one write?
Predict
Expect about 40 ms per call without TCP_NODELAY on setups where delayed ACK applies (Linux often shows this on the second small write), and well under 1 ms with it. Note that results vary by operating system, so report what you see.
Run it
1Install uv once. It fetches Python and the lab's packages for you.
Terminal curl -LsSf https://astral.sh/uv/install.sh | sh2Get the script. It is the whole lab: read it, then download it.
View the codeHide the code
nagle-plus-delayed-ack-reproduce-the.py # /// script # requires-python = ">=3.10" # /// # Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT # https://thedataengineering.com/lab-code/LICENSE.txt """Nagle plus delayed ACK: reproduce the 40 ms stall. A client sends a request as two small writes (header, then body) and waits for a short reply. Round-trip time with Nagle on (the default), with TCP_NODELAY, and with header and body in one write. Results depend on the operating system's delayed-ACK behaviour; report what you see. """ import argparse, json, platform, socket, statistics, threading, time p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) p.add_argument("--calls", type=int, default=200, help="Calls per variant.") p.add_argument("--host", default="127.0.0.1", help="Server address (another machine shows the effect more reliably).") p.add_argument("--json", help="Also write the results to this JSON file.") args = p.parse_args() def server(): srv = socket.socket(); srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) srv.bind(("0.0.0.0", 0)); srv.listen(4) def loop(): while True: conn, _ = srv.accept() def handle(c): while True: data = b"" while len(data) < 104: # 4-byte header + 100-byte body chunk = c.recv(4096) if not chunk: return data += chunk c.sendall(b"ok") threading.Thread(target=handle, args=(conn,), daemon=True).start() threading.Thread(target=loop, daemon=True).start() return srv.getsockname()[1] port = server() rows = [] for name, nodelay, one_write in (("Nagle on, two writes", False, False), ("TCP_NODELAY, two writes", True, False), ("Nagle on, one write", False, True)): c = socket.create_connection((args.host, port)) if nodelay: c.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) rtts = [] for _ in range(args.calls): t = time.perf_counter() if one_write: c.sendall(b"HDR!" + b"x" * 100) else: c.sendall(b"HDR!"); c.sendall(b"x" * 100) c.recv(2); rtts.append((time.perf_counter() - t) * 1000) c.close() rows.append({"variant": name, "median_ms": statistics.median(rtts), "p99_ms": sorted(rtts)[int(len(rtts) * 0.99) - 1]}) print(f"{name:<26} median {rows[-1]['median_ms']:7.3f} ms p99 {rows[-1]['p99_ms']:7.3f} ms") print(f"OS: {platform.system()} {platform.release()}") if args.json: with open(args.json, "w") as f: json.dump({"tables": {"variants": rows}}, f)3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal uv run nagle-plus-delayed-ack-reproduce-the.py uv run nagle-plus-delayed-ack-reproduce-the.py --help # every knob
| Knob · default | What it changes |
|---|---|
| --calls200 | Calls per variant. |
| --host127.0.0.1 | Server address (another machine shows the effect more reliably). |
Our run
200 calls on Linux loopback
| Variant | Median (ms) | p99 (ms) |
|---|---|---|
| Nagle on, two writes | 43.4 | 50.7 |
| TCP_NODELAY, two writes | 0.063 | 0.134 |
| Nagle on, one write | 0.037 | 0.1 |
Nagle on, two writes median 43.403 ms p99 50.694 ms
TCP_NODELAY, two writes median 0.063 ms p99 0.134 ms
Nagle on, one write median 0.037 ms p99 0.100 ms
OS: Linux 6.10.14-linuxkitWhat we found
On Linux, the two-write request took 43.4 ms per call with Nagle on, the delayed-ACK stall, as expected. With TCP_NODELAY it took 0.063 ms, and with header and body in one write, Nagle still on, 0.037 ms. On macOS loopback the same script showed no stall at all, so this lab's result depends on the operating system as the chapter warns.
Read it
Zero-copy, Nagle's algorithm, connection pooling and TLS termination have no semantic trade-off at the level of a record; they are constant-factor optimizations you look up. This one is worth knowing by symptom: a request-reply protocol whose latency is a suspiciously round 40 ms. Write each message in one call, or set TCP_NODELAY on latency-sensitive sockets; most RPC libraries already do.