Shared fate
Why is availability capped by the most-shared component, however many replicas you run?
Correlation is created by anything shared, and the list is longer than it looks.
| Kind | Examples |
|---|---|
| Physical | a power supply, a top-of-rack switch, a rack, an availability zone, a fibre path, a cooling system |
| Logical | a deploy artifact, a configuration value, a feature flag, a certificate, a secret, a container image, a library version |
| Dependency | DNS, authentication, the metadata service, a clock source, a package registry |
| Operational | one team, one pipeline, one runbook, one on-call person |
Logical correlation has grown while physical correlation shrank
The common wisdom is a generation out of date. Physical correlation has been engineered down over thirty years: rack diversity, zone separation, multi-path power and anti-affinity placement (a rule that keeps replicas on different machines) are now defaults. Logical correlation has been engineered up over the same period, because automation makes global, uniform change cheap, and uniformity is exactly the absence of independence.
The modern correlated failure is a configuration push, a certificate expiry or a bad deploy that reaches every cell in four minutes, because the pipeline was fast and the canary window (the trial period on a small share of servers) was short. Certificate expiry deserves its own mention as the perfect correlated failure: an identical expiry time on every instance, global clock agreement thanks to NTP, and a failure mode that is silent for a year and then simultaneous.
The availability ceiling
If R replicas each fail independently with probability p, the joint failure probability is p^R (the same arithmetic, and how correlation breaks it for copies of data, is durability's). But if all R replicas share a component that fails with probability q, then joint failure is bounded below by q, regardless of R:
- availability of a component every replica depends on
So the availability question is never "how many replicas?" It is "what is the least-available thing all of them depend on?" The answer is often a control plane, a configuration store or a single deployment pipeline.
What does this system assume that it cannot verify? That its replicas fail independently. Nothing in the system can check this, because independence is a claim about the physical and organizational world, not about the code. The graph that governs failure is not documented anywhere either. The only reliable ways to obtain it are runtime tracing and deliberate fault injection (chaos testing), the only instrument that reveals correlation you did not design.
Efficiency opposes independence. Every mechanism that lowers cost by sharing, standardizing or automating is a mechanism that raises correlation.
- Shared pools save resources and transmit failure.
- Standard images and configurations reduce operating cost and make every host fail identically.
- Automation makes a change cheap, and therefore makes it global.
- Multi-tenancy improves utilization and creates noisy-neighbor and poison-pill coupling.
Predictive form: for any efficiency measure, name what it made uniform. That is the thing that will fail everywhere at once. The corollary: independence is always purchased, with idle capacity, with version diversity, with staged rollout or with partitioned resources, and a system with no such spending has no independence, whatever its replica count says.
Correlated retries
For correlated retries and synchronized clients the mechanism is the thundering herd, and jitter is the answer. What is added here is only a source of synchronization: when an outage is the clock, correlation is perfect by construction. Every client that was independent before the incident is in lockstep after it, because the incident is the event they all observed.