Statistical checks: from predicates to estimators

Why do freshness checks deserve first place, why measure drift by effect size rather than p-values, and what do learned monitors really add?

All checks so far give a verdict per record. Statistical checks give a verdict per batch by comparing a summary (a count, an average, a quantile) with a baseline. That changes the logic completely. A predicate can be decided. Comparing an estimate needs a threshold. A threshold needs an assumption about how the numbers are distributed. And nobody has that assumption.

Freshness and volume: the checks that guard absence

Freshness is special, and deserves first place. It is the only check that fires on the absence of data. Every record-level check needs a record to look at. A pipeline that silently produces nothing passes a record-level suite with a perfect score. That is exactly what happened with Ravi's empty partner file. Freshness ("when did data last arrive?") and volume ("how many rows arrived?") are the only defences against the empty-output failure, which is both the most common serious data incident and the one most often configured as a mere warning instead of a gate.

If you take one operational conclusion from this section, take this: the checks that guard absence deserve higher authority than the checks that guard content.

Volume checks fail for a precise reason: business data is multi-seasonal. Day of week, month-end, quarter-end, holidays and campaigns all change volume. A naive "±20% around the trailing mean" band fires on every Sunday and every month-end by construction.

The fix is a conditioned baseline: compare Sunday with other Sundays, or use a seasonal decomposition. The price: the baseline needs history, the history must itself be clean, and you are now running a small forecasting system inside your quality system.

Drift: distances between distributions, built from sketches

Drift means the shape of the data has changed (for example the price distribution in a column). Measuring it needs a distance between two distributions. The inputs must be per-batch summaries that are comparable across batches, computed in the pass you are already making, and bounded in memory. That is exactly the sketch contract. So the tools are quantile sketches (t-digest, KLL), distinct-count sketches (HyperLogLog) and heavy-hitter sketches (Space-Saving).

One artifact, two consumers. The statistics that the database optimizer keeps (histograms, number of distinct values, min and max, null counts) and the baselines that a quality system keeps are the same artifact with two consumers. The optimizer computes them for query planning. A catalog stores them. Most quality stacks then compute them again independently for monitoring. This duplicate work persists because the two consumers sit in different parts of the organization, not because the artifacts differ.

The correction: significance testing is the wrong tool

Significance tests (a p-value from a Kolmogorov–Smirnov test, a t-test) are widely used for drift and are the wrong tool here. Statistical power grows with sample size. At data-engineering volumes, the p-value of any distribution test goes to zero for arbitrarily small real differences. A KS test on ten million rows will reject "no change" for a shift that no human would call a change. The p-value is measuring your row count, not your data.

The right instrument is effect size against a threshold chosen for business meaning. Examples are a population stability index (PSI) band, a percentage shift in a quantile, or an absolute change in null rate. The threshold is justified by what downstream consumers can tolerate, not by a significance level.

This does not change with the number of rows. A common rule of thumb reads PSI below 0.1 as "stable" and above 0.25 as "large shift", but your own threshold should come from what your consumers can tolerate. A KS test on the same shift with 10 million rows would report a p-value near zero, and would raise an alarm for a change nobody cares about.

Derived expectations: you encode the bug that was already there

Derived expectations means profiling the data and generating constraints from it. This is the exact mirror of inferring a schema from data, and it fails the same way: you freeze the present state, including its existing defects, and produce an expectation with no recorded intent. Six months later the check fires and nobody can decide whether it is right, because deciding needs to know what the author meant to protect.

So profiling should generate candidates. A constraint without a stated reason cannot be debugged, and checks that cannot be debugged are eventually disabled wholesale. Expectations go stale at roughly the speed at which the business changes. An unmaintained suite therefore converges on noise, and then on being ignored. The observed life cycle of every large expectation suite follows from this alone.

Learned monitors: what they actually contribute

The claim that machine-learning monitoring finds "unknown unknowns" needs correction. A learned monitor detects deviation from the historical behaviour of the metrics you chose to compute. So it cannot detect:

• a defect that was always present (there is no deviation);
• a defect that does not move any monitored aggregate.

Worse, a baseline learned from contaminated history learns the defect as normal. The monitor's blind spots are created by the defects you most need to find. (This is the principle from Propositions about the world again: the expectation is data, and here it is data derived from the very thing it is supposed to police.)

What these systems truly contribute is threshold-setting at a scale humans cannot reach. Nobody hand-tunes twelve thousand seasonal bands. A model that learns the day-of-week and month-end pattern per series solves the variance problem above at a scale manual configuration cannot. That is a real and valuable capability. But it is threshold automation, not an upgrade in detection capability. And it inherits the multiple-comparisons arithmetic in full, because it is cheap to deploy broadly.

An honest admission that weakens the "you never measure recall" principle. In practice most organisations measure neither precision nor recall. Measuring precision needs people to review every alert and record the verdict, and almost nobody keeps that up. So the accurate statement is: recall is structurally unmeasurable, while precision is merely usually unmeasured. The second can be fixed with discipline. The first cannot.

Compression: learned monitors automate thresholds, not detection. A baseline learned from dirty history is blind exactly where you need it.