Trust boundaries: where checks belong

Where should checks go, and why is the middle of a pipeline the least informative place?

The placement question comes first. Without a principled answer, checks get applied everywhere equally, which is wasteful.

The cost of a defect is roughly:

Detection latency is how long the defect lives before someone notices. Fan-out is how many consumers read the bad data. Checking has a price too: a scan at every place you check. So a check pays off most where defects are most likely to appear. And defects appear where authority changes hands, not where data merely moves.

The reason is organizational, not physical. Inside a pipeline you control, a transformation is deterministic code. If it is wrong, it is wrong on every run, and the right instrument is a test over a small fixed input, not a runtime check over production volume. Re-checking your own deterministic output at every step is testing the same code twice: once at O(1) (a small fixture) and again at O(n) (all production rows). At a boundary with a party you do not control, the other side's behaviour can change without notice, silently, with no version bump. There a runtime check buys information that no test can.

So:

The middle of a DAG is the most over-instrumented region in most stacks and the least informative.

Two more points shape every decision here. First, every check is compute inserted into the pipeline's hot path, so check depth trades directly against pipeline latency (this is Ravi's 40% slowdown). Second, every mechanism here is a position on one choice: fail fast (reject the data and stop) or fail degraded (continue, with a warning or with part of the data). Also, the time of binding decides where a defect shows up: if you check at write time, it surfaces in the producer; if you never check until read time, it surfaces in the tenth downstream consumer.

Compression: check where authority changes, test where code changes.