What validation forbids
- No validation check establishes truth. It establishes non-refutation against an expectation, and the expectation is an artifact with an author, a version and a rate of decay.
- Per-value checks verify form but never existence. The ceiling is hard and is reached cheaply, which is why coverage numbers in quality tools are systematically misleading.
- Absence cannot be detected by any record-level check. So the empty-output failure is invisible to most check suites.
- You cannot guarantee completeness and stay available under partial failure. Quarantine and block are the two horns, and there is no third.
- No single gate level is correct for several consumers with different cost asymmetries.
The escapes, and their prices
| Escape | Price |
|---|---|
| Move the referent outside (reconciliation, The reconciliation ladder) | moving or hashing the other side's data |
| Move validation upstream to the producer | highest leverage per check, but costs organizational negotiation instead of compute, which is why it is rare |
| Sample instead of scanning | detection probability for rare defects. The arithmetic is unkind: a defect rate of 10⁻⁴ needs tens of thousands of sampled rows to see it even once (The reconciliation ladder) |
| Auto-derive expectations | intent: you get checks nobody can adjudicate |
| Quarantine | invisible bias |
| Block | availability, plus permanent headroom |