A dead node looks like a slow one

What changes when one authority must be manufactured from machines that share nothing but messages?

Think about a single database engine on one machine. It has one log, one transaction manager, one memory manager. Even when such a system uses several processes, one authority exists that says what is true. Caches are copies of one origin; the origin decides.

This mechanism is where that single authority is manufactured from many machines. Every other topic assumes that someone has done this work. Here we do it.

The constraint that appears here and nowhere before: there is no shared memory, no shared clock, and no reliable way to distinguish a dead node from a slow one. The only primitive is a message that may be delayed, duplicated, reordered or dropped. And every fact a node holds about another node is at least one one-way network delay old. On a single machine you never have to act on stale information about your own hardware. Here you have no other kind.

This produces the main structural fact of this mechanism: failure stops being total and becomes partial. On one machine, a process dies and, after restart, replays its write-ahead log (a file where each change is recorded before it is applied); the failure is "stop" and the recovery is local. In a cluster, the normal condition is that some component is unavailable, to some observers, for an unknown time, and the system must keep answering anyway.

Here is the point that will be hard to unlearn: a timeout expiring is not evidence that a node is dead. It is evidence of the absence of evidence that it is alive. That is a different and much weaker thing. The whole safety apparatus of this mechanism exists because that inference is unsound.

What stops mattering. Everything below the round trip. A same-rack RTT of about 100–250 µs equals roughly 10⁵ to 10⁶ CPU instructions, or about one or two NVMe (fast SSD) reads. Once a mechanism needs a message, details such as CPU cache lines, branch prediction, memory allocation and even page faults vanish from the cost model. The cost function of this mechanism has exactly two terms: how many round trips, and how many machines must be reachable. This coarse view is why this mechanism can be reasoned about as a protocol (a set of rules for messages) instead of as a performance problem.

The governing question: who is allowed to decide, and how does everyone else learn the decision without asking?