Sampling: cheapest exactly where it is weakest

Why is sampling weakest exactly where defects are costliest?

The arithmetic is unforgiving. To see a defect of prevalence p at least once with confidence 1 − β, you need roughly

Verifying a sampled row means checking it against the world, not against another table: the invoice behind a payment, the shelf price behind a price row, a person who confirms an address. It is the only proxy in this mechanism that touches what "right" refers to, which is why it costs the most per row. Read the formula backwards and a clean sample makes a claim: if n verified rows show no defect, the defect rate is below about 3 ÷ n, with 95% confidence. Three hundred clean rows say "under 1%". They say nothing about 10⁻⁴.

Now add the usual relation between prevalence and severity: the expensive defects are the rare ones. Systemic breakage is caught by anything, even a glance. The single mispriced trade, the one account with a wrong identifier, the few rows in the wrong currency: these cost money, and they are exactly the defects that uniform sampling will not find at any budget you would approve.

Two escapes

1. Stratify. Split the data along whatever dimension defects correlate with (source, region, code path, date) and sample each stratum. A rare global defect becomes a common local one inside the affected stratum.
2. Sample by value, not by row. Where the stakes are money, verify enough rows to cover a target fraction of the total exposure. Value distributions are usually heavy-tailed (a few very large items hold most of the total), so a few hundred rows can cover most of the money. This is monetary-unit sampling, borrowed from financial audit practice. It is the right default whenever the consequence of a defect scales with a value in the row. Uniform row sampling is the right default only when every row is equally important, which is rarely the case.

Illustration. Among 800 million payment rows, suppose the largest 2,000 rows hold 55% of the money. Checking those 2,000 rows (0.00025% of the rows) covers 55% of the exposure. A uniform sample of 2,000 rows would cover about 0.00025% of it.

Compression: sampling finds defects at a rate of about 1 in n, and severity usually grows as 1 over prevalence, so sample by exposure, not by row.