What coordination forbids

What this article forbids

Telling a slow node from a dead one. This is permanent, and everything else follows from it. A silent node and a slow node look the same from outside.
Deterministic consensus in an asynchronous model with one crash fault. This is the FLP result (named after Fischer, Lynch and Paterson). "Asynchronous" means message delays have no upper limit. FLP says no algorithm can promise that it always reaches agreement in that model if even one node may crash. It is escaped only by assuming eventual synchrony (the network is slow at times but sooner or later behaves) or by randomization.
Mutual exclusion over an unmodified resource. A lease (a lock that expires on its own) leaks at the boundary. Only a token that the resource itself checks can close the gap.
A safe quorum over a membership set that the participants disagree about. This is split brain (two groups each believing they are in charge) by arithmetic. No network partition is needed.
Majority availability inside two symmetric failure domains. With only two sites, you need a third.
Global simultaneity. No "same instant" exists across machines. It is replaced by a consistent cut (a set of local snapshots that no participant can tell apart from a single instant). This is observationally equivalent to simultaneity, and nothing more.
Order, knowledge of termination, or non-idempotent payloads over gossip. Gossip spreads facts without ordering, never says "everyone has it now", and may deliver a message twice.
A coordination-free implementation of a non-monotonic invariant. This is the CALM theorem. It is the only real theorem about when you are allowed to skip all this machinery.

Escapes and what each costs

An escape is a mechanism that works around a limit. No escape is free. Each one buys something, costs something, and the cost lands on someone. The last column is the one to read twice.

Escape	Buys	Costs	Paid by
Quorum intersection	safety with no timing assumption	⌈(N+1)/2⌉ reachable members, always	availability during a partition
Leader + term	free ordering, one-round-trip commits	a single write bottleneck and a failover window	every writer, and the tail during elections
Randomized timeouts	symmetry breaking with no coordinator	a slower expected election	liveness, slightly
Lease	converts a round trip into a local decision	safety now depends on bounded clock drift	correctness, if the bound breaks
Fencing token	removes the timing assumption from safety	the resource must check it	whoever owns the resource
Gossip	dissemination with no required node	no order, no completion, tombstones forever	freshness, and a janitor
Indirect probing (SWIM)	tells a link failure from a node failure at O(1) cost	k extra messages per suspicion	a little bandwidth
Witness member	intersection cardinality without storage	one more failure domain to operate	the operator
Unaligned snapshot	removes the straggler from the critical path	snapshot size and recovery time	storage and restart
Bounded-error clock	time becomes a safety primitive	you wait out the bound	latency, explicitly
Restate the invariant (CALM, escrow, UUIDs)	removes the coordination entirely	only available when the semantics permit	nothing, which is why it is underused

The last row repeats a pattern seen in other parts of the platform: the escapes with no real cost (declare your intent, reformulate the operation) are the ones practitioners reach for last. Here the pattern is sharper, because CALM tells you exactly when the escape is available, instead of leaving it to taste.

Case, resolvedConstructed case · Ingrid at Tixora

12 seats sold twice, and the lock worked

Twelve seats were sold twice after a partition leader froze for 15 seconds past its lock's TTL, and growing the metadata cluster from 3 to 5 nodes in one step briefly produced two leaders. Where it started: Deciding without observing →

Ingrid writes the double sale as a short timeline.

The old leader froze for fifteen seconds. The lock service could not tell "frozen" from "dead" (forbidden item 1), so when the 10-second lease ran out it granted the lock to a new leader in a higher term.
The old leader woke up. It still believed it was the leader, because it had checked its lease before the pause and did not check again before acting (forbidden item 3). The lock service with a TTL could not help: a lease leaks at the boundary.
The old leader sent purchase writes for 12 seats. The seat store accepted them because it did not check anything. The fault was not the pause or the election. The fault was a resource that did not check a token.

The fixes use the part's escapes directly:

Fencing token: every write to the seat store carries the epoch (term) number of the leader that sent it. The store remembers the highest epoch it has seen and rejects anything lower. The old leader's writes now fail, whatever its clock says.
Membership change, one node at a time: when the metadata cluster grew from 3 to 5 nodes in one step, two disjoint majorities existed for a moment (one of 2 out of 3 old members, one of 3 out of 5 new members), so two leaders appeared with no network failure. Changing membership by one member per step guarantees that any old majority and any new majority overlap in at least one node.
Two data centres, automatic failover: Carmen's design with two symmetric sites cannot give majority availability in either (forbidden item 5). Ingrid adds a tiny witness member in a third availability zone. It stores no seat data and exists only to supply the vote that breaks the tie.

She also writes two questions on the team's wall: does the seat store really check the epoch on every write path? and is the membership set the same on every node? Those are the two things she cannot verify by thinking, so she adds a test for each.