Cross-field and cross-table rules

Why are cross-field rules cheap, and why does a cross-table check have a false-alarm floor?

Cross-field rules: cheap and under-used

Cross-field rules are checks across fields of the same record: "if status = 'delivered' then delivered_at is not null"; "end_time is not before start_time". They are free, because they sit in the same record and the same parse. They are the highest-value-per-cost checks in this whole mechanism, and they are chronically under-used. The reason: they require domain knowledge, while profiling tools produce range and null checks automatically. So the tools produce the cheap, weak checks, and people forget the cheap, strong ones.

def check_record(r):
    errs = []
    if r["status"] == "delivered" and r["delivered_at"] is None:
        errs.append("delivered without a delivery time")
    if r["delivered_at"] and r["delivered_at"] < r["placed_at"]:
        errs.append("delivered before it was placed")
    if r["items_total"] != sum(l["price"] * l["qty"] for l in r["lines"]):
        errs.append("order total does not equal sum of its lines")
    return errs

Cross-table rules: asynchrony makes a false-alarm floor

Cross-table rules are a different thing. They are joins, and they inherit everything joins have: cost, skew (some keys much more common than others) and, the point that matters here, timing.

Take "every order references an existing customer". This is a statement about a consistent snapshot of two datasets. Suppose orders and customers are loaded by two independent pipelines on different schedules. For part of every day, the assertion is false for reasons unrelated to data quality: the order has arrived, but its customer row has not yet landed. The check is correct, the data is correct, and the verdict is a false alarm.

So the false-positive rate of a cross-dataset check is bounded below by the asynchrony between the two datasets. You cannot tune it away with thresholds. This is Ravi's first symptom.

Correct data, failing check

There are exactly three escapes. Each has a specific price.

Escape	How it works	Price
Pinned consistent snapshot	Evaluate both sides at the same version (needs storage that supports versioned or time-travel reads).	You validate a state that may be hours older than what consumers read.
Allowed lateness	Use watermark semantics: assert only on records older than the expected skew.	Defects in recent data go undetected for exactly that window, which is the window in which they are cheapest to fix.
Pending state	Route violations to a "pending" set and re-check them later.	You have built a second pipeline with its own backlog and its own owner, and it will be the least maintained thing you own.

There is a deeper point here. Most checks assume that the datasets they compare are synchronised. That assumption is made almost everywhere and is true almost nowhere. What is new is that here the false assumption sits in the measuring instrument and not in the system being measured.

Compression: a cross-dataset check inherits the datasets' skew as an irreducible false-alarm rate.