Static stability and the control-plane asymmetry
Why must the data plane keep working without the control plane, and the recovery path not depend on what failed?
This is an easy-to-overlook mechanism, and it is among the most important defences against correlated failure.
Control planes and data planes have systematically different reliability profiles, and the difference is structural, not a matter of care.
| Control plane | Data plane | |
|---|---|---|
| Volume | low | high |
| Complexity | high | simple |
| Change frequency | frequent | rare |
| What one operation touches | every resource | one request |
The control plane will be less available, always
So any hard dependency of the data plane on the control plane caps data-plane availability at the control plane's. That is the shared-fate ceiling above in its most expensive form. The required inversion is static stability: the data plane keeps operating on last-known-good state when the control plane is unavailable, and defaults to not changing instead of trying to converge. Examples:
- A load balancer keeps routing to its current host list when service discovery is down.
- An authorizer serves cached decisions.
- A DNS resolver serves stale entries past their TTL instead of returning "no such name".
Failover planning
The pattern extends to failover planning, where it becomes counter-intuitive. If your zone-failure response is to autoscale in the surviving zones, your recovery depends on the control plane at the exact moment the control plane is handling a correlated, fleet-wide event: its own worst hour. The statically stable alternative is to pre-provision 2× capacity and take no action on failure. You pay for idle capacity in order to remove a dependency from the recovery path. This is "costs are conserved" with the price tag visible: money buys the removal of a coordination dependency.
Recovery paths must not depend on the thing that failed
The general rule: recovery paths must not depend on the thing that failed. The pathological form is the circular boot dependency:
- the deploy system that needs the object store to deploy the fix to the object store;
- the authentication service that needs to authenticate in order to restart;
- the monitoring that runs on the cluster it monitors.
These are invisible in steady state and discovered exactly once, during the incident, when they convert a 20-minute outage into a 10-hour one.