What makes a check expensive
Why is a check's cost set by the scope of state it must consult?
What decides how expensive a check is: the scope of state it must consult
Validation is derived from one question: what can be decided using only data we already have, and how much data must we have in hand to decide it?
The answer organizes the whole first half of this mechanism. The cost of a check is not set by how complicated its logic is. It is set by the scope of state it must consult. That scope maps one-to-one onto the operations a query engine already uses (scan, shuffle, join).
| Check scope | State required | Cost | Marginal cost |
|---|---|---|---|
| Per-value | the decoded value | none: rides on parsing | about 0 |
| Per-record | other fields of the same record | none: rides on parsing | about 0 |
| Per-batch | the partition or time window | one pass plus a sketch | about 1 scan, bounded memory |
| Per-dataset | all rows of this table | an index or a shuffle | an exchange |
| Cross-dataset | another table | a join | exchange plus a second scan |
| Cross-time | historical baselines | stored state plus retrieval | storage plus a consistency problem |
Everything in The free tier and Beyond one record is a derivation of what happens at each rung of this ladder, and why the ladder bends where it does.