Service discovery and load balancing
Why is the client the last to learn a change, and why does greedy load balancing herd?
Service discovery: the client is structurally the last to know
Membership and discovery are two things: membership is agreed and its staleness costs an invariant, discovery is cached and its staleness costs a retry. This section is the client half.
Convergence after a membership change is the sum of four terms:
- detection (someone notices the change);
- agreement (the cluster commits it);
- propagation to the registry (the lookup service);
- the client's own refresh.
The last term is both the largest and the only one outside anyone's control. A 30-second DNS TTL (the time a cached answer stays valid) dwarfs a 200 ms election.
It gets worse because connections are pinned. A client that holds an open HTTP/2 connection to a decommissioned server does not look the name up again, because resolution happens at connect time and requests happen afterwards. So long-lived connections and load balancing are in direct conflict. This is why max-connection-age exists as a setting. It is a forced cache expiry disguised as a connection option, and it is the only reason a fleet of clients converges after a deploy.
The design consequence is the one reached for shard maps: do not try to keep clients current. Make the destination reject on arrival and carry the correction in the rejection; anything whose staleness can cost an invariant needs a fence at the receiver.
Where the map lives
There are three choices, with a clean structure.
| Extra hop | Map copies to converge | New failure domain | |
|---|---|---|---|
| Client-side load balancing (client holds the map) | none | O(clients) | none, but the map is everywhere |
| Proxy / server-side load balancing | one round trip, always | one | the proxy is on the data path |
| Control-plane push (xDS-style) | none | O(clients), but pushed, not polled | the control plane, off the data path |
(xDS is the family of APIs, used by Envoy and gRPC, in which a control plane pushes endpoint lists to proxies and clients.)
The third row is the general answer, so it is worth understanding. Keep the single authoritative map of the proxy model. Keep the zero-hop path of the client model. Pay instead with a push-based control plane whose fan-out you now have to engineer. It is not free. It is a different bill, and it matches the control-plane/data-plane split.
A correction on load-balancing algorithms
"Least connections" or "least loaded" is widely treated as strictly better than random choice. Across independent clients it is anti-coordinated: every client acts on the same load signal, and discovery makes that signal stale by construction, so the clients herd onto the same server and oscillate. This is the same control-loop failure as the rebalancer, at millisecond scale instead of hours. The fix, power of two choices, is deliberately not greedy, and its arithmetic is there.
Subsetting
One more mechanism is easy to miss: subsetting. If every client connects to every server, the system holds O(clients × servers) connections. At 10,000 clients and 1,000 servers, that is 10,000 × 1,000 = 10 million connections, and the connection tables, health checking and TLS state dominate. Deterministic random subsetting means each client talks to a randomly chosen k of the N servers. This bounds the count (10,000 × k). The cost is poorer balance, and a rebalancing storm whenever the subset assignment changes. This is "a fixed number meets growth" in a place nobody looks: connection count is a fixed-width resource meeting fleet growth.
In short: discovery is a cache the client refreshes last and slowest, so correctness lives at the receiver, and any load-balancing policy that is greedy on a stale load signal will herd.