Coordination scales with parties
Coordination cost scales with the number of parties who must agree, not with the volume of data.
What it is
The new cost in this mechanism is coordination cost: it scales with the number of parties who must agree, not with the volume of data. Ten thousand rows read by forty teams is a harder modeling problem than ten billion rows read by one team. This is the first layer across the platform where N is people.
From Data Modeling › A layer with no oracle
Consensus does not scale out; it scales fault tolerance. Adding members buys tolerance of more failures and costs throughput, so systems run many small groups instead of one wide one (why it does not scale out).
From Distributed Systems Coordination › The arithmetic
Where it appears
In 2 mechanisms. Each says what the mechanism is about, then how this principle shows up there, in its own words.
- Distributed Systems Coordination
Who is allowed to decide, and how does everyone else learn the decision without asking?
It also explains why even cluster sizes buy nothing: N = 4 tolerates one failure exactly as N = 3 does, and costs an extra copy of everything.
Read it inConsensus
- Data Modeling
What does one row mean, who is allowed to change that answer, and what breaks downstream when they do?
The new cost in this mechanism is coordination cost: it scales with the number of parties who must agree, not with the volume of data.
Read it inA layer with no oracleConformed dimensions, the bus matrix, role-playing dimensions and reference dataEntity resolutionThe semantic (metrics) layer
Related
Made on the same pages most often.