Dead-letter versus permissive parsing

When should a pipeline reject a bad record, when may it coerce, and when must it stop?

When a record violates the rules at runtime, a pipeline has three options.

Rejection needs a quarantine store, a reprocessing path and a human who watches the queue. The third part is where it fails: an unwatched dead-letter queue is data loss with a storage bill.

The decision variable

The decision variable is not severity in the abstract. It is:

• If the consumer runs a null-rate monitor or a row-count monitor, permissive parsing is survivable.
• If they cannot, rejection is mandatory, because you are choosing between a visible gap and an invisible lie.

Permissive parsing is defensible only with a violation counter attached. The counter restores the error channel that coercion destroyed.

The rate threshold almost everyone omits

One bad record per million is a bad record. Forty percent bad is not bad data at all. It is an undetected schema change. Dead-lettering it turns a loud break into a slow bleed at full volume.

The policy needs an escalation from per-record handling to stop-the-line when the violation rate crosses a threshold. The absence of that escalation is why many incidents are found by a downstream consumer, not by the pipeline that caused them.

WINDOW, THRESHOLD = 1000, 0.02       # look at the last 1,000 records; stop above 2% bad

def handle(record, recent_bad_flags, dead_letter, publish):
    ok = validate(record)
    recent_bad_flags.append(not ok)
    del recent_bad_flags[:-WINDOW]
    if not ok:
        dead_letter.put(record)               # per-record disposition
        metrics.increment("schema_violations")  # the counter
    else:
        publish(record)
    if len(recent_bad_flags) == WINDOW and sum(recent_bad_flags) / WINDOW > THRESHOLD:
        raise PipelineHalt("violation rate above threshold: likely a schema change")

Reject when the consumer cannot notice the damage, coerce only with a counter, and escalate on rate, because a 40% dead-letter rate is a schema change, not bad data.