Propositions about the world

Why can no system prove its data is right, and how do validation and reconciliation differ?

CaseConstructed case · Ravi at Freshlane

14 sets of checks, and a price 100 times too big

After a bad week, Ravi's manager says: "Add data checks to every step of the nightly pipeline." Ravi adds a block of checks after each of the 14 steps. The nightly run now takes 40% longer. Then, on a Tuesday, one partner supermarket changes its price file from rupees to paise (1 rupee = 100 paise) without telling anyone. Every price becomes 100 times bigger. None of the 14 sets of checks notices, because each check only compares step output with step input, and both are consistently wrong by the same factor. Anita finds the problem on Wednesday, when the finance report shows 100 times more revenue from that partner than usual.

Why did 14 sets of checks miss it, and why are they so slow? We need to understand what kind of thing a data check is.

How it ends →

Every layer of a data system below this mechanism can check itself. A checksum (a number computed from the bytes) decides whether a disk block has rotted, with no outside help. A write-ahead log (a file where changes are recorded before they are applied) decides whether a write landed. A two-phase commit (a protocol where all participants vote before a change is final) decides whether all participants agreed. A type check decides whether a field decodes as an integer. In each case the question is internal: it is a statement about the system's own state, and the system holds enough information to answer it.

This mechanism is about the first kind of question that is about the world. "This customer ID is real." "Yesterday's revenue is plausible." "These two rows are the same person." None of these can be decided from the bytes. The thing the statement is about lives outside the system. So the mechanism has to change shape. You cannot prove the content correct. You can only fail to disprove it against an expectation that some person wrote down. That expectation is itself an artifact: it has a maintainer, a version, and a rate at which it goes out of date.

That is the break. The new constraint is the absence of an internal ground truth. The constraint that stops mattering is the determinism of the verdict. Below this mechanism, a failed check has exactly one meaning and one response: retry, roll back, or fail over. Here, a failed check has three possible meanings, and you cannot tell which from the failure alone:

The data is wrong.
The world changed legitimately (a new product category, a new status code, a sale day).
The expectation is stale (the check itself is out of date).

Recovery machinery in a fault-tolerant system assumes the system already knows what "correct" was. This mechanism is where that knowledge is manufactured.

So the governing question becomes:

How much of a claim about the world can be tested using data we already hold, what does the test cost, and who pays when it is wrong in each of its two possible directions?

And the sentence to remember:

The two halves are structurally different

This mechanism splits into validation and reconciliation. The division is not cosmetic. The two halves differ in what supplies the reference.

Validation compares data to a stated expectation: a constraint, a range, a distribution, a prior belief. The reference is a human assertion. Validation is cheap, single-pass and local. But it can only catch violations that someone anticipated. Its failure mode is silence about the unimagined.

Reconciliation compares data to a second independent authority: the source system, a parallel computation, or an earlier version of the same dataset. The reference is another system's answer. It catches things nobody anticipated, because it needs no hypothesis: a difference announces itself. Its cost is that you must move or hash the other side's data. Its ambiguity is that a disagreement never tells you which side is wrong.

Validation is a hypothesis test. Reconciliation is a differential (a comparison of two answers). You need both, because each covers the other's blind spot. The choice between them is mostly economic: hypothesis tests are cheap and narrow, differentials are expensive and broad.

You measure precision but never recall. Fired checks are visible and can be judged. Defects for which no check was written produce no evidence of their own absence. Every quality programme overestimates its coverage, and the bias is structural, not a lack of rigour. The only instrument that partly escapes it is a differential against an independent authority, which is why reconciliation exists as a separate discipline instead of being "more validation".