Differential observability

Why do health checks lie, why do deep checks correlate, and who should observe health?

The structural claim: a health signal generated by a component, using that component's own resources, measuring that component's own view, is a claim about the process, not about the service. Clients experience a different quantity: the full path, including the network, the queue, the dependency, and the disk that is now answering reads in 2 seconds.

Why cheap checks lie

The failures are ordinary and follow from the check being cheap, which is the property you asked for:

• A health endpoint served from a dedicated thread pool reports healthy while the request pool is exhausted.
• A check that returns a constant reports healthy while every query times out.
• A check that never touches the disk reports healthy on a disk doing 40 IOPS.

In each case the component is telling the truth about what it measured. It measured the wrong thing. The gap is exactly the gap between "the process is running" and "the service is being delivered".

Self-reports measure the component, not the service. Only an external observer measures the service, and the two diverge exactly when it matters. Health endpoints, exit codes, job status, control-plane state and success metrics are all self-reports. The divergence is not a bug to fix: the component genuinely cannot observe its own delivered service, because delivery includes the path to the client. Predictive form: for any health signal, ask whether the observer is inside or outside the boundary it certifies. If inside, assume a gray-failure mode exists in which it reports green, and name it. The monitoring system is no exception: a metrics pipeline that drops 30% of its data is undetectable by construction, because the instrument that reports health is the one that is broken.

Deep checks create correlation

Now the tension, which has no clean resolution. Make the check deep: exercise the real path, the real dependency, the real disk. You have now created the correlation mechanism of shared fate. When the shared dependency degrades, every instance fails its check at the same time, and the orchestrator ejects the entire fleet, turning a partial degradation into a total outage. Deep checks manufacture correlated failure; shallow checks miss gray failure. Both are real and both are common. This was Selin's second mistake.

The practical answer: bound the action

The practical resolution is not a better check but a bound on the action: never eject more than a fixed fraction of a fleet (an eviction cap), whatever the checks say, and above that fraction assume the checker is wrong, not the fleet. Fail open on mass unhealthiness. This is a statement about epistemics written as a policy: when the evidence says everything is broken, the most probable explanation is that the evidence is broken.

def evict_unhealthy(nodes, is_healthy, max_fraction=0.2):
    bad = [n for n in nodes if not is_healthy(n)]
    if len(bad) > max_fraction * len(nodes):
        alert("mass unhealthy: assuming the checker is wrong, evicting nobody")
        return []
    return bad

Change who observes

The stronger move is to change who observes. Health determined by clients and not by the component measures the quantity that matters, by construction. This is what outlier ejection in load balancers and client-side circuit breaking do. The cost is that a client's view mixes the server, the network and the client itself. That leads to the one reliable attribution tool in this article, a table with two axes: how many clients report, and about how many servers.

Neither axis alone identifies anything. The cross-product does. Any health system that aggregates in only one way (server self-report, or per-client thresholds) is structurally unable to tell these four cases apart.