What gray failure forbids

Why is a correct failure detector impossible, and which escapes remain?

What this article forbids

It forbids a correct failure detector. Not a good one, not a well-tuned one: a correct one, in an asynchronous system, as a matter of logic. Therefore every automated failover is a bet that can be wrong. That is why fencing exists: it makes a possibly-still-alive old leader harmless. The system must remain correct when the detector is wrong, because it will be wrong.

It forbids trusting self-reported health for anything that matters. And it forbids assuming that a green control plane means a working data plane: they are separate observations of separate quantities, and the whole failure mode is the case where they disagree.

Escapes and costs

Escape	Buys	Paid in
Watchdog / suicide on missed deadline	Decidability: manufactured fail-stop	Capacity destroyed; false positives
Client-observed health, outlier ejection	Measures the quantity clients care about	Mixes server, network and client without cross-tabulation
Peer-relative outlier detection	Catches gray failure below absolute thresholds	Blind to correlated degradation; needs similar work across peers
Eviction caps / fail open on mass unhealthiness	Stops detector error from becoming a total outage	Knowingly serves from degraded hosts
Hedging	Random stragglers	Extra load; useless against deterministic ones
Data validation as detector	The only gray-failure detector in the data plane	Compute cost; maintained expectations that drift

Case, resolvedConstructed case · Selin at Wayhop

One slow node, and every search slow

One of 200 search nodes ran 100 times slower with a failing fan while its health check stayed green, and Wei's nightly job exited 0 with a city missing from the report. Where it started: Gray failure →

The slow search node. The node had a failing fan and ran 100 times slower. A dead node would have cost 1/200 of capacity and affected 0% of searches. The slow one cost the same capacity and affected 100%. Selin adds client-side outlier detection (a node whose p99 is above 3× the median of its peers is ejected, using peer comparison) and an absolute alert for the case where all nodes are slow.
The mass eviction. Her deep check made all 200 nodes fail together when the database slowed. She keeps the deep check but adds an eviction cap of 20%: if more than 40 nodes fail the check, nobody is evicted and a human is paged, because the likelier explanation is a broken checker or a shared dependency. She also adds the client-by-server table to the dashboard.
Wei's missing city. The nightly job exited with 0, because "exit code zero" only says that the control flow finished. A late partition meant one city was never read. Wei adds reconciliation checks as the pipeline's failure detector (expected cities, row counts against the same weekday a week before, freshness), and a counter for every row the parser coerces to null. A masking mechanism without a counter hides failures.

Bruno summarizes: "So we do not know which of our machines are bad, and the machines that pass are not the machines that work."