Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

State has mass

Why are partitioning and replication different tools, and what arithmetic does every distributed layout divide by?

The rest of this mechanism is about deciding: who leads, who is alive, what the current configuration is. Decisions are weightless. A decision that turns out wrong can be re-made for the cost of a message. This article is about the one thing here that has mass: data.

The constraint that governs here: state must be copied byte by byte over a finite link, and during the copy it exists in two places at once while exactly one of them must be authoritative. The first half makes movement slow: hours, not milliseconds. The second half makes movement dangerous, because ambiguous ownership of mutable state is not degraded service, it is corruption. Everything in this article is a consequence of these two facts colliding.

Two independent ideas generate the design space, and mixing them up is the standard error:

  • Partitioning divides. It buys capacity and total throughput. It does nothing for availability: if a node is lost, its partitions are lost outright.
  • Replication multiplies. It buys availability, durability and read capacity. It does nothing for write throughput: it is a fixed R× multiplier on every write.

Every real system uses both. So the real unit of distribution is the pair (partition p, i), which is currently owned by some node n. The map from these pairs to nodes is the most consequential metadata in the cluster. Most of the difficulty in this article is not in choosing a good map, but in changing one while every client and every node holds a stale copy of the old one.

; the hard problems all live in the map that connects them, because that map is mutable state that every participant and no participant can be forced to refresh.

The arithmetic everything divides by

Let D be the data size in bytes, N the number of nodes, P the number of partitions and R the replication factor.

  • . Partition-replicas per node = P·R/N.
  • Write work per node is proportional to R·w/N (w is the total write rate); read work is proportional to r/N, served from R candidates. So aggregate write capacity scales as N/R and aggregate read capacity as N. This asymmetry makes every replication decision a bet on the read:write ratio.
  • Recovery after losing a node must re-replicate D·R/N bytes. If the surviving replicas of those partitions are spread over many nodes, the sources work in parallel, and the transfer takes roughly D·R/(N²·b), where b is the spare bandwidth of one node. Recovery time therefore falls as 1/N². This is the real argument for many small partitions: partition count converts a linear recovery into a quadratic one, and recovery time is what durability depends on (rebuild time is the real durability variable).
  • Cross-region floor. Fibre gives about 10 µs/km for a round trip, so New York to London is about 70–80 ms, whatever you build. A key whose writes must cross that link one after another is capped at about 13 writes per second (1 s ÷ 75 ms ≈ 13). This is the only number in this article that no engineering will ever improve.