A gate is a check with authority

Which actions can a failed check take, and why did dataset-level gating need a pointer swap?

CaseConstructed case · Ravi at Freshlane

1,000 orders set aside, and revenue 4% low

Ravi's team now has gates. Two incidents follow.

A new check quarantines 1,000 of 100,000 orders in one batch because they fail to parse. The batch is published with the remaining 99,000. Nobody is paged. Two days later, Anita sees that the "average basket size" has dropped by 3% and total revenue is about 4% below the payment provider's number. She cannot find any error.
The recommendation team asks Ravi to block the pipeline when any check fails. A real defect stops the pipeline for six hours. When it restarts, it must process seven hours of input. The warehouse is slow for a whole day, and the finance report is late. At the same time, the finance team says a blocked report is a better outcome than a wrong one.

The same gate is right for one consumer and wrong for another. Why?

A check with no authority is a signal: it reports (how a signal is routed and who it wakes is observability's question). What makes something a quality gate is the ability to stop something. That is the whole distinction. It is about authority, not about the measurement.

The four-level action space

The action space has four levels, not two:

Level	What it does
Warn	Record the failure. Notify nobody.
Quarantine	Set the offending records aside. Publish the rest.
Block	Publish nothing.
Restate	Publish a correction over something already released.

The level is a property of the check, chosen by a cost comparison: block when the cost of bad data downstream is greater than the cost of no data downstream.

For a regulatory submission, blocking is clearly right.
For a recommendation feed, stale-but-present is better than absent, and blocking is self-harm.

The cost comparison is per consumer, but the gate sits at the producer. A single dataset that feeds a compliance report and a dashboard cannot have a correct gate level for both.

Restate: correcting what was already read

The first three levels act before anyone reads the data. Restate acts after: the wrong value has been published, and some readers have already used it. Detecting the defect is worth little without a defined way to reach them. A correction needs three things.

The old answer must still be addressable. If publication overwrote it, nobody can see what changed, or check what an earlier report was built on. Keeping each published version (a table version, a snapshot, an as-of read) is what makes "before" and "after" comparable.
The readers must be found. Who read the wrong version is a lineage and usage question: the downstream tables built from it, and the reports and people that queried it during the window.
The correction must say what kind it is. A silent fix replaces the value and tells nobody. That is right for a dashboard that everyone re-reads every morning. A published correction tells each reader what changed, from when, and by how much. It is required wherever the number has left the team: a finance report, a regulator, a customer. Choosing between the two is the same cost comparison as the gate level, made per consumer.

Recomputing is other mechanisms' work. How a downstream result takes back a value it already emitted is a retraction, and re-running the affected intervals is a backfill. What belongs here is the decision: which readers must be told, and what the corrected dataset promises about its own history.

The coupling problem: why consumers re-check

This is why consumer-side checks multiply. Each consumer defends itself independently. That duplicates work and, worse, makes the consumers diverge on definitions, so that two teams end up disagreeing about whether yesterday's data was good. This is an organizational constraint expressing itself as a technical artifact. No tool resolves it. Only an agreed contract between producer and consumers does (what the data promises, who owns it, what happens on failure).

Why dataset-level gating became practical: publication as a pointer swap

One mechanism deserves a causal claim. Dataset-level gating became practical when publication became a pointer swap. It matters because a data deploy cannot be rolled back the way a code deploy can: once a write lands, consumers have read it. The only workable rule is to make visibility the final, atomic step.

The pattern is write-audit-publish. First you write the data to a hidden place. Then you audit it. Then you make it visible all at once. Structurally it is a two-phase commit: the write is the prepare, the audit is the vote, the pointer swap is the commit. This needs a state in which data is fully written but not visible, and then an atomic switch to visible.

On a plain file system used as a table, that meant writing to a staging directory and then moving directories. Moves are not atomic across many files, and they are not cheap.
Modern table formats (formats that keep a table's file list in metadata) make the current version of the table a metadata pointer, switched by compare-and-swap (an atomic "change this pointer only if it still has the value I last saw"). Now "validate, then publish" is an O(1) commit decision.

The validation pattern followed the storage primitive, not the other way around.

Streaming has the opposite situation. There is no batch boundary, so there is no publication point, so there is no place for a gate. Streaming validation is necessarily per-record routing (good records go forward, bad ones go aside) plus windowed statistics, and its only dataset-level tool is a circuit breaker on the topology (stop the flow). That, not culture, is the real reason batch and streaming quality designs look so different: streaming has no commit point.

Compression: a gate is a check plus authority. Gates live at producers, but the cost comparison that sets their level lives at consumers.