Chaos testing as mechanism verification

What does deliberately injecting failures verify, and what can it never prove?

Every recovery mechanism is a claim: the standby will take over, the retry will converge, the backup will restore, the fallback will serve. These are claims about rarely executed paths, made by people, verified by nothing. The base rate for such claims is poor, for a structural reason: the recovery path has no users generating feedback, so a bug in it has no discovery mechanism other than the disaster itself.

Chaos testing (deliberately injecting failures into a running system) is the discovery mechanism. Its logic is amortization of discovery. You will pay for exercising the recovery path exactly once per latent defect: either at 2 a.m. under full load with data at risk, or at 11 a.m. on a Tuesday with a blast radius you chose. Chaos does not reduce the number of defects. It moves the time when you find them. This is the "costs are conserved, only moved" principle in a domain where the move is almost free.

Three boundaries (the practice is often oversold)

1. It can only find failures you injected. It is a test, not a proof. It has no coverage guarantee over a failure space that is infinite and contains modes nobody has named. Its record is strongest on failures that are easy to inject (kill a process, drop a network link, fill a disk). It is weakest on the failures that cause major outages, which are correlated, gradual and configuration-driven. Fail-slow is the notable gap. Injecting latency and packet loss teaches far more than injecting death, yet it is done far less often, precisely because the fail-stop model that makes everything here work is the model that reality violates.
2. It requires the observability to notice. Injecting a failure into a system that cannot tell you what happened produces an outage, not an experiment. Chaos without monitoring is vandalism.
3. Its real product is the second-order finding. A single-component injection mostly confirms what you expected. What chaos really finds are dependencies nobody declared: the "optional" service that turns out to be on the critical path, the failover that works but takes eleven minutes, the fallback that calls the thing that is down. Would chaos still find anything if the dependency graph were accurately known? Probably yes, because the interesting failures are in timing and capacity, not in topology. That is a prediction, not a result.

The method

The method is the one in this article (Hypotheses and variables), with four parts:

1. State a steady-state hypothesis in observable terms. Not "the system stays up", but "p99 of hold-writes stays under 100 ms and error rate under 0.1%".
2. Inject one fault. For example, kill one node, or add 200 ms of delay to one network link.
3. Declare the abort condition before you start. For example, "stop if the error rate exceeds 2%".
4. Bound the blast radius, so that the experiment cannot become the incident.

The hypothesis must be falsifiable (it must be possible for the result to prove it wrong). That is why "kill a node and see what happens" is exploration and not an experiment: nothing could refute it.

What only fault injection can find is correlated failure and gray failure, which are invisible to every other method here. A gray failure is when a component is slow or partly broken and not clearly down. Performance measurement assumes components are either up or down, but the expensive failures are the ones where a node is slow and not dead, or where three services secretly share a dependency that nobody mapped. You cannot discover an undocumented shared dependency by reading the architecture diagram, because if it were on the diagram it would be documented.

What chaos adds is one claim: