Replication factor and quorums as a survivability budget

How many nodes can be lost before writes stop, and what does each extra copy cost in latency?

A quorum system keeps N copies of each item. A write must reach W of them and a read must ask R of them. Elsewhere the same N, R, W describe what a read means. Here we read them as a budget: how many nodes may be gone before the system stops.

The cost is simple, and that is the point

N copies cost N× storage and N× write bandwidth, forever, to survive N−1 failures. The cost is linear and so is the survivability. No mechanism beats that ratio, so the only real lever is what counts as a copy: erasure coding for the bytes, and a witness for the vote (Witness and tiebreaker replicas). For availability, erasure coding has one side effect worth knowing here: a degraded read (one piece is missing) must fetch from k nodes, which turns a durability mechanism into a tail-latency mechanism the moment anything is down.

Quorum overlap has two jobs

• R + W > N guarantees that any read set overlaps the last write set, so the read sees the latest write. This is the consistency reading.
• W ≤ N − f guarantees that writes can still proceed with f nodes down. This is the availability reading.

Both rules constrain the same three numbers, so choosing a quorum is choosing a point on the trade-off directly, with no slack.

Why N is almost always odd, and why 3

A majority quorum needs 2f + 1 nodes to tolerate f failures. Even N buys nothing: four nodes tolerate one failure exactly as three do (a majority of 4 is 3, so two failures stop it), at 33% more cost. Going from f=1 to f=2 (five replicas) is an economic choice and a judgement about correlation. Five-replica systems exist for control-plane metadata, where losing the data loses the whole cluster, and almost nowhere else.

Placement and rebuild time

The number of failures a quorum tolerates assumes the failures are independent and that a lost node is rebuilt before the next one goes. Both are durability's numbers, and both are worked through there: three copies in one rack are approximately one copy (correlated failure), and loss needs f+1 failures within the rebuild window (rebuild time). That is exactly what happened to Mediflow's rack: two copies shared a domain, so they counted as roughly one.

The latency consequence people forget

A W-of-N write waits for the W-th fastest response. So write latency is an order statistic of the replicas' latency distribution, not an average. Raising W moves you further into the tail, and the tail is fat (a few very slow responses). This is why synchronous cross-region replication is priced in round-trip time (RTT) and often abandoned. It is also why the dial of synchronous, semi-synchronous and asynchronous replication is a direct price list: durability is bought with acknowledgment latency, and the exchange rate is the shape of the replicas' latency tail. Lower W (asynchronous) cuts latency but sets RPO to the replication lag, which is largest during the very failure you wanted protection from.