P-values measure your row count

Simulate a modelNeeds Python with NumPy, SciPy

Problem

A statistical test asks whether a difference could be chance, and with enough data almost nothing can be. On a large table, a drift test's p-value says more about the row count than about the size of the change: a shift too small to matter becomes "highly significant" at ten million rows. A measure of effect size, such as the Population Stability Index (PSI), which compares the share of rows in each quantile bucket, stays about the same as n grows and stays small when the change is small.

Draw samples from a normal distribution with mean 100 and standard deviation 15. Compare a sample with mean 100.1 (a 0.07% shift) to a baseline, using scipy.stats.ks_2samp, at n = 10³, 10⁴, 10⁵, 10⁶, 10⁷. Also compute the PSI over 10 quantile buckets.

The question. For the same tiny shift, how do the KS p-value and the PSI change as the sample grows from a thousand to ten million rows?

Predict

At ten million rows, what does the KS test say about a 0.07% shift?

Not significant (p above 0.05)BorderlineOverwhelmingly significant

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

p-values-measure-your-row-count.py42 linesDownload

View the code

p-values-measure-your-row-count.py

# /// script
# requires-python = ">=3.10"
# dependencies = ["numpy", "scipy"]
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""p-values measure your row count.

A sample with mean 100.1 against a baseline with mean 100 (sd 15), a 0.07%
shift, compared with the two-sample Kolmogorov-Smirnov test and with the
Population Stability Index over 10 quantile buckets, for n from 10^3 to 10^7.
"""
import argparse, json
import numpy as np
from scipy.stats import ks_2samp

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--shift", type=float, default=0.1, help="Shift of the sample's mean.")
p.add_argument("--max-exp", type=int, default=7, help="Largest n is 10^max-exp.")
p.add_argument("--seed", type=int, default=0, help="Random seed.")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()
rng = np.random.default_rng(args.seed)

def psi(a, b, buckets=10):
    edges = np.quantile(a, np.linspace(0, 1, buckets + 1)); edges[0], edges[-1] = -np.inf, np.inf
    pa = np.histogram(a, edges)[0] / len(a); pb = np.histogram(b, edges)[0] / len(b)
    pa, pb = np.clip(pa, 1e-6, None), np.clip(pb, 1e-6, None)
    return float(np.sum((pb - pa) * np.log(pb / pa)))

rows = []
for e in range(3, args.max_exp + 1):
    n = 10 ** e
    a = rng.normal(100, 15, n); b = rng.normal(100 + args.shift, 15, n)
    pv = ks_2samp(a, b).pvalue
    rows.append({"n": n, "p_value": float(max(pv, 1e-300)), "psi": psi(a, b)})
    print(f"n = {n:>10,}   KS p-value {pv:10.3g}   PSI {rows[-1]['psi']:.6f}")

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"sizes": rows}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal
```
uv run p-values-measure-your-row-count.py
uv run p-values-measure-your-row-count.py --help     # every knob
```

Knobs
Knob · default	What it changes
--shift0.1	Shift of the sample's mean.
--max-exp7	Largest n is 10^max-exp.
--seed0	Random seed.

Our run

KS p-value for a 0.07% shift, by sample sizeApple M2 · 8 cores · 8 GB

Data table

KS p-value for a 0.07% shift, by sample size
Rows per sample	p-value
1K	0.121
10K	0.628
100K	0.089
1M	0
10M	0

macOS 26.6.2 · Python 3.13.3 · numpy 2.5.4 · scipy 1.18.1 · run 2026-10-20

The same shift at every size

The same shift at every size
n	KS p-value	PSI
1K	0.121	0.022
10K	0.628	0.001
100K	0.089	0
1M	0	0
10M	0	0

Output · uv run p-values-measure-your-row-count.py

n =      1,000   KS p-value      0.121   PSI 0.021592
n =     10,000   KS p-value      0.628   PSI 0.001260
n =    100,000   KS p-value     0.0885   PSI 0.000264
n =  1,000,000   KS p-value   0.000189   PSI 0.000052
n = 10,000,000   KS p-value   2.37e-51   PSI 0.000059

What we found

The p-value went from 0.121 at a thousand rows to 0 at ten million, for the same 0.07% shift, as expected. The PSI was 0 at ten million, far below the 0.1 usually read as a moderate shift; it was larger at a thousand rows (0.022) only because small samples are noisy, not because the change was bigger.

Read it

On data at warehouse scale, statistical significance is guaranteed and therefore uninformative. Alert on effect size (PSI, a difference in means in business units, a share of rows outside a range) with thresholds chosen from what matters, and use p-values only where samples are small.