Scenarios

A defect's cost ≈ latency × fan-out × consequence, and defects enter where authority changes: the partner files, the app events and the payment file (ingress), and the finance table itself (egress). Test the transformation steps with fixtures in CI instead of re-scanning at every hop: with 200 million rows and 6-minute scans, 14 hops cost 84 minutes, while 5 boundary scans cost 30.

• Freshness and volume on each input, the guard against absence, with volume baselines conditioned on the weekday.
• A median price per category compared with yesterday, which catches unit changes such as rupees to paise.
• At egress, reconciliation against the payment provider, as an independent authority.

The finance report blocks on failure, since late is cheaper than wrong; publish the quarantine rate by region. Blocking needs spare capacity to catch up afterwards, about 1.5 times the stop time at 60% utilization.

The recall of any validation suite cannot be measured; only the independent comparison partly escapes that.

Count the checks. With 12,000 of them at a false-positive rate of 0.001 each, expect 12 false alarms a day by construction; at 0.01, 120. Alert fatigue is designed in, not a failure of discipline.

1. Choose a small set of gates that may page, say 40, by impact on consumers.
2. Demote the rest to dashboard signals.
3. Replace significance tests with effect sizes against business thresholds.
4. Condition volume baselines on seasonality.
5. Require an owner and a reason for every check, so a firing check can be judged: every expectation ages.
6. Measure precision by recording a verdict on each alert for a month.

Each of these choices gives something up. And confirm that freshness and volume checks are gates, since the missed incident may well have been empty output.

The difference in load timing sets a floor on false alarms. With orders loaded every 10 minutes and customers hourly, a first order can wait about 50 minutes for its customer row.

• A pinned snapshot of both tables: exact, but validates a state hours old, and needs versioned reads.
• Allowed lateness, say 70 minutes: simplest; defects in the newest 70 minutes are noticed 70 minutes late, which is when they are cheapest to fix anyway.
• A pending state: nothing is lost, but you run a second pipeline, with a backlog and an owner.

Start with allowed lateness, paired with an inferred "unknown customer" member so late arrivals are counted rather than lost; consider a snapshot only for the finance gate.

The check can never be better than the synchronization of its inputs.

Ask whether the failures are random. For example: 1,000 of 100,000 orders are quarantined, averaging ₹1,950 against ₹450 for the rest. The published average is ₹450 against a true ₹465, 3.2% low, and 4.2% of revenue is missing, from 1% of rows. If failures correlate with region, app version or order size, the remainder is biased.

• Finance: block, or publish with the quarantined share of revenue shown if it exceeds a threshold such as 1%.
• A dashboard: quarantine, and show the rate along every dimension it uses.
• A recommendation feed: warn, and keep serving the previous data.

Quarantined records soon become impossible to reprocess as schemas and code move on, so set a review deadline of a few weeks and treat older ones as an archive.

There is no settled definition of whether such a run "succeeded", and almost no dataset publishes its completeness in a form consumers can read.

All pairs ≈ n² / 2 = 2 × 10¹⁴ comparisons; at a million a second, about 6.3 years on one core. Blocking is required.

Several independent blocking passes (postcode area, phone prefix, phonetic surname), keeping candidates near a billion pairs per pass (20 million records in 200,000 blocks of 100 give 4,950 pairs each). A comparator that scores name, address and phone. Clustering that avoids plain transitive closure: require two independent signals, and cap cluster size, sending clusters above 20 to review.

Map (source, source ID) to an entity ID, with the version of the matching rules, so a wrong merge can be undone. It costs storage, and a join on every read.

Recall is capped by blocking, so measure it on a labelled sample; measure precision by reviewing a sample of merges. Watch for collapse when thresholds are loosened.

Climb the ladder: daily row counts; sums per day and region; a hash per partition (a sum of 64-bit hashes of normalized rows, with exact decimals, not XOR); a Merkle-tree walk inside the partitions that differ; then a cell-by-cell diff of the few rows left.

Locating d = 40 differences among n = 10⁸ rows costs about 40 × 27 hashes × 32 bytes ≈ 35 KB, instead of moving 80 GB.

Canonical form: rounding hides sub-paisa errors, so state the rule. Floating-point sums depend on order, so use decimals. The source keeps changing, so compare both sides at a common cut, such as a watermark.

A difference does not say who is wrong: designate the provider as the system of record for settled amounts. If the gap comes from writing to both systems separately, replace that with a single writer and derive the other from its log; the audit then becomes a weekly check.

It detects deviation from the history of the metrics you chose. It cannot see defects that were always present, or that do not move a monitored aggregate; trained on dirty history, it learns the defect as normal.

Thresholds set automatically, at a scale people cannot manage (thousands of seasonal bands). But it inherits the arithmetic of multiple comparisons: more monitors, more false alarms.

Use it to set thresholds on a limited set of high-value series; keep human-owned gates for absence and finance; add comparisons against independent sources for recall; and measure precision by recording verdicts.

The recall of any validation system cannot be measured, by its structure.

n ≈ −ln(β) ÷ p = 3.0 ÷ 10⁻⁵ = 300,000 rows for 95% confidence; for 99% (−ln 0.01 = 4.6), 460,000. A uniform sample is costly and nearly useless for a rare but expensive defect.

Sample by value (monetary-unit sampling): weight rows by how much money they carry, and check the top ones; if a few thousand rows hold most of the money, check them all. Stratify by source or currency field, so a rare global defect becomes common within one stratum.

A cheap rule on every record: the currency must be in the allowed set, and the amount within that currency's usual range. It costs nothing at parse time and beats sampling for this defect.

One definition per feature; an offline store (columnar, append-only, with valid and availability times) for training; an online key-value store (one row per entity) for serving.

Batch SQL feeding both stores is simple but allows timing skew. Computing once in the streaming path and writing to both closes it, but loses batch SQL as the language for definitions.

The difference in freshness between the stores, paging above a threshold (for example 15 minutes).

Drift in the input distribution needs separate monitoring.