Quarantine: how a loud failure becomes a quiet bias

How does setting bad records aside turn a loud failure into a quiet bias?

Partial failure forces a choice with no clean option.

• Reject the whole batch for one bad row in a billion, and a trivial defect causes a total outage.
• Accept the good rows and set the bad ones aside, and you have published a dataset whose completeness claim is now false, by an amount nobody downstream can see.

The second failure is the more dangerous one. The reason is not that data is missing. It is that the missing data is not missing at random. Bad records correlate with something: one source system, one region, one text encoding, one client version, one date range. So the quarantined set is a systematically biased sample, and every number computed over the published remainder is biased in the same direction. A count is low. A regional average leans towards the regions that parse cleanly. Nothing in the output says so.

Quarantine converts a missing-data problem into a biased-data problem. That is strictly harder to detect, because it produces plausible numbers.

Only 1% of rows were quarantined, yet 4.2% of revenue is missing. The defect is not random: it removed the biggest orders.

The mitigation is not technical sophistication but exposure. Publish the quarantine rate as a first-class property of the dataset, next to the data, at the same granularity at which the data is read. If a consumer aggregates by region, they need the quarantine rate by region. Anything less hides the bias instead of recording it.

This is also a case of the rule that costs are conserved, never removed: quarantine did not remove the cost. It moved it from the pipeline owner (who would have been paged) to a downstream analyst (who will not be, and who will be quietly wrong in a meeting).

The dead-letter queue and the poison pill

Two structural notes.

First, the fate of a dead-letter queue is universal and predictable. It grows, nobody owns it, and its contents become unreprocessable. Reprocessing needs the schema version and code version that existed at capture time, and both have moved on. A DLQ has a short half-life of reprocessability. Treat anything older than a few weeks as an archive, not a backlog.

Second, compare with a poison pill in messaging systems. A poison pill is a record that kills the processor (it crashes the consumer each time it is read). That is a transport and robustness problem with an exact solution (skip it, send it aside, fix the parser). A quarantined record here processes perfectly well and is simply wrong. That is a semantic problem with no exact solution. Same queue, different disease.

Compression: quarantine trades an outage for an invisible bias, because bad records are never a random sample.