Summary and key takeaways
The break. Below this mechanism, a system can prove its own state correct. Here, claims are about the world. A check can only fail to refute a claim against an expectation someone wrote down, and a failed check has three meanings: the data is wrong, the world changed, or the check is stale.
Two ladders, complementary blind spots
| Validation | Reconciliation | |
|---|---|---|
| Reference | a stated expectation | a second independent system |
| Strength | cheap, single-pass, local | needs no hypothesis; differences announce themselves |
| Weakness | silent about the unimagined; the expectation ages | costs data movement; never says which side is wrong; only as good as the independence of the second side |
| Cost driver | scope of state consulted (value, record, batch, dataset, cross-dataset, cross-time) | localisation demanded (count, aggregate, hash, partition hash, Merkle tree, cell diff) |
Mechanisms in one line each:
- Check where authority changes (ingress, egress). Test where code changes.
- Distributed constraints stay as optimizer claims nobody checks: a false claim gives wrong answers silently.
- Type checking is free because decoding is not optional. Its ceiling is form, not existence.
- Null with four meanings should become explicit values, so it can be counted.
- A cross-dataset check has a false-alarm floor equal to the asynchrony of its datasets. Escapes: pinned snapshot, allowed lateness, pending state.
- Freshness and volume guard absence. They deserve more authority than content checks.
- At data-engineering volumes, p-values measure your row count. Use effect size against a business threshold.
- Alert fatigue is arithmetic: 12,000 checks × 0.001 = 12 false alarms a day.
- Four actions: warn, quarantine, block, restate. The level depends on the consumer, but the gate sits at the producer.
- Quarantine turns missing data into biased data, because bad records are never a random sample. Publish the quarantine rate.
- A circuit breaker is paid for in advance as spare capacity. Break by partition.
- Fuzzy matching is two stages. The cheap blocking stage fixes the recall ceiling. Transitive clustering collapses at a percolation point. Keep merges as a reversible overlay.
- Find d differences among n rows with about d · log n bits, not O(n). Use a commutative combiner (sum, not XOR), canonicalise on purpose, and use exact decimals.
- Audit as a star around a system of record. Most audits pay for a dual write.
- A data diff has complete recall over its own change. Golden fixtures regenerate into nothing.
- Learned monitors automate thresholds, not detection.
- Sampling needs n ≈ −ln(β) / p rows. Sample by exposure (value), not by row.
Three principles that came from the mechanisms:
- Every threshold is a classifier with two error types. Alert fatigue and silent corruption are one dial.
- You measure precision but never recall. Coverage is always overestimated.
- Every assertion about data is itself data with an owner, a version and a rate of decay.
Three principles from the reconciliation half: a differential detects but never blames; a cheap prefilter sets the recall ceiling; locating a difference costs d · log n, not n.
Formulas to keep:
| Formula | Meaning |
|---|---|
| defect cost ≈ detection latency × fan-out × consequence | where checks pay off |
| n ≈ −ln(β) / p | sample size to see a defect of prevalence p with confidence 1 − β |
| catch-up time = (u × T) ÷ (1 − u) for utilization u, stop time T | why a stop is cheap at 60% and costly at 90% |
| pairs = n(n − 1) / 2 | why blocking is needed |
| k(k − 1)/2 versus k − 1 | mesh versus star audit |
Unresolved: partial-failure semantics and correction propagation have no settled answer.