Which way the error points
What it is
Why the approximation is safe here. A Bloom filter's false positives let a few non-matching rows through. But an exact join follows the filter, so those rows are removed later. False positives cost work, not correctness. This is the general safe pattern: an approximate pre-filter in front of an exact operator gives an exact answer at reduced cost. Compare approximate aggregation in Aggregation algorithms: spending the bound before data moves, where the approximation is in the answer itself. Same mathematics, very different risk.
From Query Processing › Distributed joins and filters
Data skipping has a "safe direction" (it may read a block it did not need, never skip one it did), and that is the sharpest contrast with the planning articles. Cardinality estimation had no safe side, so its errors were unbounded. Data skipping has a safe side, so every engine adopts it aggressively. Nobody has ever been paged because a zone map (a stored min/max summary of a block) was too conservative.
From Query Planning & Optimization › Changing how much data exists
Each stage is a cheaper test than the next. Each stage has one-sided error: it can only produce false positives ("maybe there") and never false negatives ("not there, but you said yes" is allowed; "there, but you said no" is not). Each stage is ordered by cost per unit of selectivity (how much data it lets you skip per unit of effort).
From Core Data Structures › Compositions
Where it appears
In 14 mechanisms. Each says what the mechanism is about, then how this principle shows up there, in its own words.
- Core Data Structures
How do we choose a data structure when the real cost is set by the hardware, not by Big-O?
Each stage has one-sided error: it can only produce false positives ("maybe there") and never false negatives ("not there, but you said yes" is allowed; "there, but you said no" is not).
Read it inPutting the pieces together: the five currencies and the four compositionsSpending correctness: what a sketch isMinHash, SimHash and LSH
- Query Processing
Why can one question be answered by plans whose costs differ by a factor of ten thousand, and what does each operator force the engine to do?
This is the general safe pattern: an approximate pre-filter in front of an exact operator gives an exact answer at reduced cost.
Read it inBroadcast, shuffle and runtime filtersSet and dedup operations: aggregation with an empty accumulator
- Query Planning & Optimization
How does a system choose a plan for a question when the facts that decide the best plan are unknown until the plan has run?
Data skipping has a safe side, so every engine adopts it aggressively.
Read it inChanging how much data existsThe reframe: a layer limited by what the system can know
- Memory & Resource Management
Who is allowed to hold how much memory, for how long, and what happens to the work when the answer is less than it asked for?
Under-estimation is the cheaper error if and only if everything is spillable.
Read it inWhere the grant number comes from
- Fault Tolerance & Delivery Semantics
What is the cheapest thing that can be safely done twice?
A Bloom filter used for dedup fails toward dropping a legitimate record, not toward admitting a duplicate.
Read it inIdempotency: the three sources
- Scheduling & Workload Management
Which work runs now, which work waits, and who pays for the waiting?
> Route on cheap evidence, and bias the classifier toward the failure mode you can afford.
Read it inCost-based workload routingPrioritization and short-query lanes
- Table & Storage Maintenance
Who pays for the gap between the order data arrives in and the order it is read in: the writer, every reader, or a background job, and what is the break-even?
Moving an object to a cold tier is a prediction problem with asymmetric costs, made before the access is observed.
Read it inLifecycle tiering
- Data Modeling
What does one row mean, who is allowed to change that answer, and what breaks downstream when they do?
The threshold is an asymmetric business risk, not a tuning parameter.
Read it inEntity resolution
- Schema Management & Evolution
What happens when the shape of your data changes on its own schedule, and nobody can change the readers, the writers and the old data at the same moment?
So tightening a constraint is always verify, then declare, in that order.
- Data Quality & Validation
How do you check that data is right, when "right" is a claim about the world and not about the bytes?
So the false-positive rate of a cross-dataset check is bounded below by the asynchrony between the two datasets.
Read it inCross-field and cross-table rulesMultiple comparisons and alert fatigueDeduplication: three different problems in one word
- Metadata, Catalog & Lineage
What can a system prove about its data without reading the data, and how does it know that proof is still true?
When two stores cannot be updated atomically, order the writes so that the inconsistency that actually happens is the recoverable one.
Read it inReconciliation and drift
- Observability & Operations
What is the smallest description of a running system from which we can find a fault, fast enough to matter, at a cost we can still pay while the system is failing?
Page on symptoms the consumer experiences, not on every cause that might produce them.
Read it inAlerting is a base-rate problem, not a detection problemSLOs, error budgets and burn rate
- Specialized Processing Models
What do you build when a query predicate cannot be reduced to a range over an ordered key?
The recall of a multi-stage system is capped by stage one.
Read it inPhrase queries and faceting
- Measurement & Experimental Method
What would have to be true for this number to mean what I think it means?
Stop when the next run cannot change the decision, and fix that count before you start.
Related
Made on the same pages most often.