Variance costs more than volume
Variance costs more than volume.
What it is
- Conservation: what arrives must leave or wait. This is Little's Law. It is bookkeeping, and it cannot be false.
- Variability: arrivals and service times fluctuate. Fluctuation plus a limited server makes queues even when average capacity is enough. Most painful performance problems come from variability, not from lack of throughput.
Earlier parts said predictability beats work (CPU) and pattern beats volume (network). This article gives the mathematics: **variance is the enemy, and averages hide it.
From Physical Substrate & Performance Laws › Little's Law and queues
Queues are not caused by load. They are caused by variability meeting load. (If one customer arrives exactly every 60 s and each takes exactly 59 s, nobody waits. Add randomness to either, and waiting appears.)
From Physical Substrate & Performance Laws › Little's Law and queues
Hashing concentrates skew by design. Every row of a hot key (a key with a huge number of rows) lands in exactly one bucket. So the finishing time of the join becomes the finishing time of one task that holds that key. This is the variance rule with real teeth: the average bucket is fine, the variance kills you. And unlike most skew, you cannot fix it by adding parallelism, because the partitioning function forbids splitting that key.
From Storage: Physical Organization › Partitions and zone maps
Where it appears
In 25 mechanisms. Each says what the mechanism is about, then how this principle shows up there, in its own words.
- Physical Substrate & Performance Laws
Why do trade-offs exist at all?
Branchless code does a small amount of guaranteed extra work for every item and removes the variable, large cost of mispredictions.
Read it inThe core and the operating systemLittle's Law and the queueing wallTail at scale, and why percentiles do not combineTCP, kernel bypass and the latency tiers
- Data Movement
How do records cross between two systems that have independent rates, independent availability and independent lifetimes, and what are you willing to lose when the assumption underneath the crossing breaks?
The correct way to think about sizing: a buffer should be sized for absorbing bursts, not for sustained overload.
Read it inBufferingWhat changed: the epistemic limitPartitioning: the four strategiesTwo kinds of skew: different problems, different fixes
- Persistence & Durability
When a system says "yes, your data is safe", what exactly has it promised, and what did it have to do to keep that promise?
This is the rule that variance costs more than volume, and here it is the only reason the textbook design is used almost nowhere.
Read it inCheckpointing: sharp, fuzzy, incrementalLog truncation and retention policiesJournaling: the filesystem's own WAL
- Core Data Structures
How do we choose a data structure when the real cost is set by the hardware, not by Big-O?
(Variance law.) The mitigations (rate limiting, compaction scheduling, pacing) all convert a bursty background cost into a smooth one: pay a little in the mean to buy a lot in the tail.
Read it inHash tablesLSM-trees
- Encoding, Serialization & Compression
What do we turn our values into when we turn them into bytes, and what does that choice cost?
The cost of the block is set by its maximum, so 0.8% of the data (1 of 128 values) decides 100% of the width.
Read it inDelta and frame of reference
- Indexing
When does a separate "where is it?" structure pay for itself, and what does it cost to keep it true?
Skew is a cost when it is unknown and unexploited, and an asset when it is statically describable.
Read it inPartial indexes: the one place skew is an assetSet membership: inverted indexes
- Transactions & Concurrency Control
What happens when many users change the same data at the same time, and how do we keep the result correct?
It is the variance, not the mean, that defeats you.
Read it inDeadlockWhy concurrency is a different kind of problemChoose a concurrency schemeChoose a distributed commit
- Query Processing
Why can one question be answered by plans whose costs differ by a factor of ten thousand, and what does each operator force the engine to do?
The principle needs this boundary condition: concentration is expensive for operators that process rows and cheap for operators that reduce them.
Read it inThe output does not grow with the inputRange, interval and as-of joins
- Query Planning & Optimization
How does a system choose a plan for a question when the facts that decide the best plan are unknown until the plan has run?
Corollary 1: min/max is the most outlier-sensitive summary.
Read it inWhy min/max skipping works or fails completelyDiagnosing a bad plan, and what planning forbidsThe cost model and the cardinality estimatorAdaptive query executionJoin order search
- Memory & Resource Management
Who is allowed to hold how much memory, for how long, and what happens to the work when the answer is less than it asked for?
> With a heavy-tailed input-size estimate, choose the algorithm with the better tail, not the better mean.
Read it inSort spill, and choosing the operator by its tailNUMA awarenessWhere the grant number comes fromLRU and its two failure modesPage size: the granularity decision nobody revisits
- Caching
What happens when we deliberately keep a second copy of data that no protocol forces to be correct?
When miss costs vary by orders of magnitude, hit rate is the wrong objective, and optimizing it does harm.
Read it inCost-aware eviction: the policy family that can be badly wrongThe first copy with no obligationMiss penalty and the percentile you promisedSize a cache
- Distributed Systems Coordination
Who is allowed to decide, and how does everyone else learn the decision without asking?
The median fan-out request contains a p99 event.
Read it inScatter-gatherBarriers and distributed snapshots: group state without simultaneityLeader election and epochs: one mechanism, many namesReplication topologies: where is the serialization point?
- Fault Tolerance & Delivery Semantics
What is the cheapest thing that can be safely done twice?
The key property: the cost scales with the divergence, not with the size of the dataset.
Read it inAnti-entropy, read repair and hinted handoffFailover: automatic, manual, fenced
- Emergent Failure Dynamics
What happens when nothing is broken, every component is healthy, and the system's own response to stress makes the stress worse?
One line: a dead node costs 1/N of capacity and affects no requests; a slow node costs the same capacity and affects all of them.
Read it inWhy fail-slow is worse than fail-stopThe formal condition
- Scheduling & Workload Management
Which work runs now, which work waits, and who pays for the waiting?
> Parallelism cannot beat the longest path.
Read it inDAG-aware schedulingFIFO, priority and multi-level feedback
- Cost Mechanics
What is metered, and what is the smallest increment you are charged for?
When item costs are heavy-tailed, hit rate is the wrong objective.
Read it inCost-based routing and result cachingThe computable value of pruningBuying and selling demand riskIdle: the most misunderstood cost
- Time & Temporal Semantics
When am I allowed to act on data that may still be incomplete, and what do I owe the reader if I was wrong?
So every waiting decision here is a percentile decision, not an average decision, and it is sharper here than anywhere else, because the tail is so much longer than the middle.
Read it inThree clocks: which time is on the record?Watermarks: declaring completeness
- Streaming
What does it cost to remember, when the input never ends?
Triggers are timers, timers are state, and watermark jumps fire them all at once.
Read it inTriggers, evictors and the timer serviceWatermark propagation
- Data Modeling
What does one row mean, who is allowed to change that answer, and what breaks downstream when they do?
It is also an instance of the variance law ("variance costs more than volume"): the problem is not how many members there are, but that a few attributes change all the time.
Read it inAs-was or as-is: the slowly changing dimension types
- Data Quality & Validation
How do you check that data is right, when "right" is a claim about the world and not about the bytes?
Now add the usual relation between prevalence and severity: the expensive defects are the rare ones.
Read it inStatistical checks: from predicates to estimatorsDeduplication: three different problems in one wordSampling: cheapest exactly where it is weakest
- Metadata, Catalog & Lineage
What can a system prove about its data without reading the data, and how does it know that proof is still true?
This is a positive feedback loop ("rich get richer").
Read it inSearch, discovery and ranking
- Orchestration & Workflow
How do you make unreliable, unobservable, irreversible side effects behave like a pure function of a time interval?
In a series of queues with independent policies, the last queue's policy dominates.
Read it inPriority and starvation across pipelines
- Observability & Operations
What is the smallest description of a running system from which we can find a fault, fast enough to matter, at a cost we can still pay while the system is failing?
So the useful signal is the sign and size of the derivative (the rate of change), not the level:
Read it inConsumer lag and skew
- Specialized Processing Models
What do you build when a query predicate cannot be reduced to a range over an ordered key?
Synchronization substitutes for locality.
Read it inPregel, BSP and iterative algorithmsTraversal and degree skew
- Measurement & Experimental Method
What would have to be true for this number to mean what I think it means?
Report the shape, because the mean averages across mechanisms that never occur together.
Read it inRecording is irreversible; distributions, not averages
Related
Made on the same pages most often.