Connection pooling, keep-alive and heartbeats

How should connections be pooled and kept alive, and why is every heartbeat timeout a guess?

Connection pooling

Opening a connection costs a TCP handshake (1 RTT), a TLS handshake (1–2 RTT) and often authentication. On a 1 ms link inside a datacenter that is a few milliseconds. Across regions it is tens to hundreds of milliseconds. Doing this for every request is pure waste when you will make thousands of requests.

Example. Cartwheel calls a service in another region with a 100 ms RTT. A new connection needs 1 RTT for TCP plus 1 RTT for TLS 1.3, so 200 ms before the first byte of the request is sent. A request that needs only 100 ms (one round trip) takes three times as long when the connection is new.

Pooling keeps established connections and reuses them. It is a nearly free win and is used everywhere. These are the parts worth knowing.

The pool size is admission control in disguise. The maximum pool size caps how many requests can be in progress against that dependency. As Flow control showed, a hard cap on concurrency is the most reliable form of overload protection. Size it with Little's Law: connections = throughput × latency.

The exhaustion policy is a load-shedding decision you probably did not make. When the pool is empty, the library can block forever, block with a timeout, or reject at once. The default in many libraries is block forever. That turns a slow dependency into a total hang of your own service, because every thread ends up waiting for a connection. Check what yours does. This is a common production surprise.

One pool per downstream, always. If one pool serves several dependencies, then a slow dependency uses up all the connections and starves the healthy ones. This is the bulkhead failure, and connection pools are where it most often happens.

Idle connections die silently. NAT tables (devices that translate between private and public addresses), firewalls and load balancers remove idle connections from their tables without telling either end. Your pool hands out a connection that looks fine, and the first write fails. There are three fixes:

• validate on borrow: test the connection before use (costs a round trip);
• keep the maximum idle time shorter than the shortest reaper timeout on your path (usually the right answer);
• retry once on a stale connection (pragmatic, but it must be safe: use it only for idempotent operations).

Pool per host versus per service. With DNS-based service discovery (the service name resolves to a list of IP addresses), a pool keyed on the service name can keep using old IP addresses after a deploy or a scale event. Refresh DNS and recycle connections from time to time. Connection pools have a maxLifetime setting for exactly this reason, and it is not only about database failover.

---

Keep-alive and heartbeats

Two different things are often mixed up.

Keep-alive means keeping an idle connection open and preventing intermediaries from removing it. There are two ways. TCP keepalive is done by the operating system, and its default is 2 hours, which is useless without tuning. Application-level pings (HTTP/2 PING frames, gRPC keepalive) are better: you control the interval, and they pass through proxies that ignore TCP options.

Heartbeat means liveness detection: "Is the other side still alive and working?" A heartbeat has meaning, and it is the source of a specific class of distributed-systems failures.

The failure-detection problem

Over an asynchronous network (one with no guaranteed delivery time), you cannot tell the difference between a node that is dead and a node that is slow. This is not an engineering gap. It is a theoretical impossibility, the same limit as the lost-ack ambiguity in From one machine to two. Every heartbeat timeout is therefore a guess, and the timeout value is a trade-off:

• Short timeout: fast failure detection, but also false positives. A garbage-collection pause or a short network problem makes the system treat a healthy node as dead and remove it.
• Long timeout: no false positives, but you keep sending traffic to a dead node for the whole timeout.

Example. A consumer sends a heartbeat every 3 seconds, and the broker declares it dead after 10 seconds without one. A 12-second JVM garbage-collection pause makes the consumer miss four heartbeats. It is evicted although it is healthy. A timeout of 30 seconds would have tolerated the pause, but would also have left a really dead consumer holding its partitions for 30 seconds before anyone took over.

The consequences of a false positive can be severe: a consumer group rebalance for no reason (the group redistributes its partitions among members, and during the rebalance consumption pauses), a leader election that splits the cluster, a node evicted in the middle of its work. Rebalance storms are the worst case. A slow consumer sends a heartbeat late, is evicted and rejoins. The rejoin triggers a rebalance that slows everyone. The slower members send late heartbeats and are evicted too, and so on. This is a metastable failure: a bad state that keeps itself going even after the original cause is gone, because the system's own reaction (more evictions, more rebalances) sustains it. Flow control described this pattern for retries. Here it appears through failure detection.

Two answers worth knowing

Separate the liveness heartbeat from the work loop. Kafka splits two timeouts: session.timeout.ms (a heartbeat thread answers "am I alive?") and max.poll.interval.ms (the work loop answers "am I making progress?"). The split exists because a consumer that is processing a large batch is alive but is not polling. If one timeout covers both, every slow batch looks like a death. If you build your own system, keep them separate.

Phi-accrual failure detection is the better general design: instead of a yes/no verdict at a fixed timeout, it outputs a suspicion level from the distribution of recent heartbeat intervals (Failure detection explains it). Cassandra and Akka use it.

Heartbeats can also carry payload. Health status, load, capacity, epoch or term numbers (a counter that identifies the current leader's era) all fit. It is a cheap channel that already exists. For control-plane decisions, using it to report load is often better than a separate metrics path.