A decision that has no answer

Why can no failure detector decide whether a component is dead or slow?

CaseConstructed case · Selin at Wayhop

One slow node, and every search slow

One of the 200 search nodes has a failing cooling fan and runs about 100 times slower than the others. Its health check says "healthy". Yet every rider search is slow, not 1/200th of them. Selin adds a "deep" health check that runs a real query. During a database slowdown, all 200 nodes fail the deep check at once, and the orchestrator evicts the entire fleet. A partial slowdown becomes a total outage.

Meanwhile Wei finds a different kind of failure. His nightly trip-events job exits with code 0, the orchestrator shows green, the dashboards load, but one city has been missing from the revenue report for a week.

In both stories, no component is "down". The failure detector (the check that decides what is healthy) is the thing that is wrong.

How it ends →

Everything so far has assumed a decidable predicate: a yes/no question that can be answered. Failover needs to know that the leader is dead. Quorums (majority votes) need to count live members. Health checks need to eject bad hosts. Cells need to know which cell is broken. Retry budgets need to know that a request failed. Every reliability mechanism is downstream of one binary question, and this article is about the fact that the question has no answer.

The constraint is logical and permanent: in an asynchronous system (one with no bound on message delay) you cannot distinguish a slow process from a dead one. There is no observation that separates them, because the only evidence you have is the absence of a message, and absence is consistent with both. Every failure detector in production is therefore a timeout: a guess with a tunable error rate, not a measurement.

What stops mattering is the fault itself. Nothing has to be down. Components are up, returning correct answers, passing their own checks, and destroying the system. What starts mattering is epistemics: who observes, what they observe, and whether what they observe is the thing that matters.

Fail-stop is not a property systems have; it is a property we pay to approximate, and every layer that improves reliability by masking errors makes the approximation worse.