Independence is assumed
Independence is assumed everywhere and is true almost nowhere.
What it is
Every mechanism assumes something it cannot verify. The question to ask is: "What does this mechanism assume that it cannot verify, and who pays when the assumption is wrong?" For clustering, the assumption is independence: that columns behave independently of each other. It is assumed everywhere and is true almost nowhere. This section is where it costs the most and also pays the most.
From Storage: Physical Organization › Partitions and zone maps
Before the arithmetic, one assumption has to be said out loud: replication, quorums and erasure coding all divide by a failure probability that is assumed to be independent. Independence is assumed almost everywhere and is true almost nowhere. What really kills clusters are correlated failures: the same rack, the same firmware version, the same bad manufacturing batch, the same bad deploy, the same operator.
From Persistence & Durability › Replication and quorums
The gap between an order-0 model and the best model of the same text is what assuming independence costs, as a measured number. An order-0 model is exactly the assumption that symbols are independent. On English text, that assumption costs a factor of four. The cost of assuming independence is not a vague risk. It is a number you can compute by comparing your model's order with the true conditional entropy.
From Encoding, Serialization & Compression › General-purpose compression
Where it appears
In 23 mechanisms. Each says what the mechanism is about, then how this principle shows up there, in its own words.
- Data Movement
How do records cross between two systems that have independent rates, independent availability and independent lifetimes, and what are you willing to lose when the assumption underneath the crossing breaks?
Independence is assumed everywhere and is true almost nowhere.
Read it inWhat changed: the epistemic limit
- Storage: Physical Organization
What should sit next to what, given that I must decide at write time and find out at read time?
Every mechanism assumes something it cannot verify.
- Persistence & Durability
When a system says "yes, your data is safe", what exactly has it promised, and what did it have to do to keep that promise?
Independence is assumed almost everywhere and is true almost nowhere.
Read it inDurability across machines: replication, quorums and erasure codingPlacement, rebuild time and erasure codingBackup and point-in-time recoveryEnd-to-end checksums, scrubbing and bit-rot detection
- Core Data Structures
How do we choose a data structure when the real cost is set by the hardware, not by Big-O?
Independence (uniform keys, uncorrelated failures, requests that do not influence each other) is assumed everywhere and is true almost nowhere.
- Encoding, Serialization & Compression
What do we turn our values into when we turn them into bytes, and what does that choice cost?
This is what assuming independence costs, as a measured number.
Read it inGeneral-purpose compression: the last passWhat lightweight encoding forbids
- Indexing
When does a separate "where is it?" structure pay for itself, and what does it cost to keep it true?
The common point: the data's own distribution is information.
Read it inWhen the map gets built: paying later, or paying less
- Transactions & Concurrency Control
What happens when many users change the same data at the same time, and how do we keep the result correct?
Independence is assumed and false, and it does not always fail in the direction you expect.
Read it inAtomicity becomes an availability liabilityWhy concurrency is a different kind of problemChoose a distributed commit
- Consistency & Isolation Models
When many users, many objects and many copies exist at once, which histories does the system promise never to show, and who writes the code for the histories it does show?
So the guarantee is weakest exactly during the failures it was chosen for.
Read it inTunable consistency
- Query Processing
Why can one question be answered by plans whose costs differ by a factor of ten thousand, and what does each operator force the engine to do?
Hash join buys O(n) with two assumptions, and every other join algorithm in the catalogue is the repair for one of them failing.
Read it inWhere the index comes from: nested loop, hash and sort-mergeSkew: the assumption of uniformity, pricedBags, NULLs and the operator latticeSet and dedup operations: aggregation with an empty accumulatorA sort produces a property, not an answerFrom what to how: why a query has many plans
- Query Planning & Optimization
How does a system choose a plan for a question when the facts that decide the best plan are unknown until the plan has run?
This is the independence assumption failing in its most painful operational form.
Read it inProjection pushdown and Top-N pushdownThe cost model and the cardinality estimatorStatisticsThe reframe: a layer limited by what the system can know
- Memory & Resource Management
Who is allowed to hold how much memory, for how long, and what happens to the work when the answer is less than it asked for?
It is "independence assumed, but false" wearing a business model.
Read it inAdmission control by memory footprintPinning and unpinning: where residency meets correctness
- Distributed Systems Coordination
Who is allowed to decide, and how does everyone else learn the decision without asking?
The boundary condition is where hedging is usually misused: hedging pays if and only if the slowness is a property of the attempt, not of the request.
Read it inHedgingScatter-gatherThe arithmetic worth carrying
- Fault Tolerance & Delivery Semantics
What is the cheapest thing that can be safely done twice?
Multiplying failure probabilities requires independence.
Read it inRedundancy cannot be amortizedReplication factor and quorums as a survivability budgetFailover: automatic, manual, fencedRetry with exponential backoff and jitterTask re-execution and blacklisting
- Emergent Failure Dynamics
What happens when nothing is broken, every component is healthy, and the system's own response to stress makes the stress worse?
> Redundancy multiplies availability only under independence, and every mechanism that makes redundancy affordable (sharing, standardizing, automating) manufactures correlation.
Read it inStructure instead of timeLoad stops being an independent variableStragglers: two regimes that need different answers
- Scheduling & Workload Management
Which work runs now, which work waits, and who pays for the waiting?
The latency hierarchy and the failure hierarchy are the same tree.
Read it inRack and zone awarenessBin packing versus spread placementCost-aware placement and spotWorkload isolation: pools, warehouses, clusters
- Cost Mechanics
What is metered, and what is the smallest increment you are charged for?
The work is AND-coupled: all parts must survive.
Read it inCross-AZ charges and placementSpot capacity: the vendor's idle stock, resold
- Streaming
What does it cost to remember, when the input never ends?
The minimum rule assumes sources advance independently and at similar speeds.
Read it inWatermark propagation
- Data Modeling
What does one row mean, who is allowed to change that answer, and what breaks downstream when they do?
This assumption is the independence law from A layer with no oracle in its purest form, and many-to-many relationships violate it directly.
Read it inMany-to-many: bridge tables, fan traps and chasm trapsEntity resolution
- Data Quality & Validation
How do you check that data is right, when "right" is a claim about the world and not about the bytes?
A differential converts "is this right?" into "do these two agree?" That is a cheaper question, a weaker one, and only as strong as the independence of the second opinion.
Read it inMultiple comparisons and alert fatigueThe reconciliation ladder: detecting is cheap, locating is what costs
- Orchestration & Workflow
How do you make unreliable, unobservable, irreversible side effects behave like a pure function of a time interval?
It is the earlier principle at its purest: independence assumed, not true.
- Observability & Operations
What is the smallest description of a running system from which we can find a fault, fast enough to matter, at a cost we can still pay while the system is failing?
Externalize the small critical path: alert evaluation, paging, and a dead-man's switch (an alert that fires when telemetry stops arriving, so silence itself is detected).
Read it inThe telemetry pipeline is a data system
- Specialized Processing Models
What do you build when a query predicate cannot be reduced to a range over an ordered key?
A structure fitted to a distribution is valid only for that distribution; a predicate that selects a sub-distribution invalidates it.
Read it inHybrid search and filtered ANN
- Measurement & Experimental Method
What would have to be true for this number to mean what I think it means?
OVAT gives you a line through your starting point and calls it a map.
Read it inPredictions, hypotheses and variablesEvaluation splits and pinned dataInference, run counts and the two variances
Related
Made on the same pages most often.