Thundering herd

Which common clocks synchronize clients, and why is jitter the only fix for the peak?

Load is a random variable, and everything downstream is provisioned against its mean (its average). Synchronization does not change the mean. It moves the mass. This is the principle that variance matters more than volume.

N independent arrivals spread over a window follow Poisson statistics: the variance equals the mean, so the peak is about mean + a few × √mean. Small.
N perfectly synchronized arrivals give a peak of N/ε for an arbitrarily small spread ε. The daily volume is the same, but the instantaneous rate is unbounded.

So the question is always: what is the common clock? There are four kinds, and they are structurally different.

Common clock	What synchronizes	Example
Wall time (cron and calendar boundaries)	Everyone agrees on when :00 is; NTP (the network time protocol) made clocks agree globally	Every nightly job starts at 00:00
Shared TTL (time to live of a cache entry)	Write time: everything filled in the same cache fill expires in the same millisecond	100,000 entries cached in one burst, all with TTL 60 s
Recovery events	The failure itself: restart, failover, deploy, partition heal. The outage is the clock, so clients that were uncorrelated before are perfectly correlated after	All apps reconnect the second the service returns
Shared library defaults	Nothing but a constant, which is enough	Every client sleeps exactly 1 s before the first retry

What exponential backoff does and does not do

Exponential backoff (doubling the wait between attempts) lowers the frequency of retry waves, but it does not touch their correlation. The herd still arrives together, just less often.

Jitter (random spread added to the wait) is the only term that manufactures the spread δ. It converts the peak N/ε into N/W, where W is the width of the jitter window.

Decorrelated jitter goes further. The rule is sleep = random(base, previous_sleep × 3). It randomizes the state of each client, so clients cannot re-converge after several rounds.

import random
def next_sleep(base, previous, cap):
    return min(cap, random.uniform(base, previous * 3))   # cap: the longest wait allowed

The cost is honest and small: some clients wait up to W longer than necessary. You buy survival of the worst case (p100) with a slightly worse p99 latency.

Livelock: the herd of two

Livelock is the same synchronization at the smallest scale. CPU is at 100%, state changes constantly, and throughput is zero. Examples: two threads that politely back off in lockstep forever; two schedulers that each undo the other's placement; retry loops that reliably conflict again. (Think of two people in a corridor who both step left, then both step right, again and again.) The signature is high utilization with zero goodput, the same gap as in congestion collapse with a completely different cause. Symmetry is the disease, so the fix is again randomization: if the two people in the corridor pick a random direction, they get past each other quickly.

Synchronization appears throughout this mechanism under many names: thundering herd, livelock, convoy, correlated retry, breaker chorus, autoscaler step, expiry storm, staggered-restart failure. There is one mechanism, with one cost (added latency), and it applies to control decisions as readily as to requests.

Selin's midnight spike. Every night at exactly 00:00 the payout job and an app refresh both fire, because cron and a shared library default both pick "on the hour". Nothing about daily volume changed; only the clock did.