The arithmetic worth carrying

How do round-trip tiers and quorum tails set what a design can do?

Coordination is priced by locality tier, and the tiers are decades apart. Same-rack RTT is about 0.1–0.25 ms, cross-AZ about 0.5–2 ms, cross-region 10–150 ms, with a hard floor of about 10 µs per km of fibre round trip. A protocol that needs one RTT per operation is fine inside an AZ, marginal across AZs, and architecturally impossible across regions. Nothing here is faster than geography. So where you place the coordination point is a bigger decision than which protocol you pick.

Consensus does not scale out; it scales fault tolerance. Adding members buys tolerance of more failures and costs throughput, so systems run many small groups instead of one wide one (why it does not scale out).

A quorum is the only fan-out that does not amplify the tail. Waiting for any 3 of 5 replicas is faster in the tail than waiting for one designated node, because the slowest replicas are not on the critical path (the arithmetic).

Replication multiplies availability only if failures are independent. Three replicas, each up with probability p = 0.99, give availability of about 0.9997 under independence, when the system needs a majority (at least 2 of 3) up: 3 × 0.99² × 0.01 + 0.99³ = 0.0294 + 0.9703 = 0.9997. That is 30 times fewer outages than a single node at 0.99 (unavailability 0.01 versus 0.0003). But three replicas in one rack give you the availability of one rack: the failure domain of the placement matters more than the replication factor (placement across failure domains).

Any fixed failure-detection timeout is either slow or wrong, because response times have fat tails: a long timeout waits, a short one suspects live nodes (Failure detection).