Dead-letter rate threshold

Simulate a modelNeeds Python

Problem

A dead-letter queue sets bad records aside so that one malformed record does not stop a pipeline. It assumes bad records are rare and independent. A schema change breaks that assumption: suddenly a large share of records are "bad" for the same reason, the queue swallows them, and the pipeline keeps going with a hole in its output that nobody notices. A rate threshold over a window turns that situation into a halt: occasional failures go to the queue, a burst stops the stream.

Simulate a stream of 100,000 records. For the first 50,000, 0.01% are invalid. At record 50,001, a "schema change" makes 40% invalid. Process it with (a) a pure dead-letter queue, and (b) a windowed threshold policy (halt above 2% over the last 1,000 records). Report how many good records each policy lost or delayed, and how long until a human would notice.

The question. How many records does each policy dead-letter or hold back, and how quickly does it react to the schema change?

Predict

Halting above 2% of the last 1,000 records: how many records after the schema change does the pipeline stop?

Under 100About a window, 1,000Thousands

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

dead-letter-rate-threshold.py48 linesDownload

View the code

dead-letter-rate-threshold.py

# /// script
# requires-python = ">=3.10"
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Dead-letter queue against a rate threshold.

100,000 records; 0.01% invalid for the first 50,000, then a schema change makes
40% invalid. (a) a pure dead-letter queue: bad records are set aside and
processing continues; (b) a windowed threshold: halt when more than 2% of the
last 1,000 records are invalid. Reports records dead-lettered, good records
processed, how long until the stream stops, and how much a human would see.
"""
import argparse, json, random
from collections import deque

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--records", type=int, default=100_000, help="Records.")
p.add_argument("--change-at", type=int, default=50_000, help="Record at which the schema change lands.")
p.add_argument("--bad-after", type=float, default=0.4, help="Invalid share after the change.")
p.add_argument("--threshold", type=float, default=0.02, help="Halt threshold (share of the window).")
p.add_argument("--window", type=int, default=1000, help="Window size.")
p.add_argument("--seed", type=int, default=0, help="Random seed.")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()
random.seed(args.seed)

stream = [random.random() < (0.0001 if i < args.change_at else args.bad_after) for i in range(args.records)]   # True = invalid
def run(policy):
    dlq = good = 0; window = deque(maxlen=args.window); halted = None
    for i, bad in enumerate(stream):
        window.append(bad)
        if policy == "threshold" and len(window) == args.window and sum(window) / args.window > args.threshold:
            halted = i; break
        if bad: dlq += 1
        else: good += 1
    held = args.records - (halted if halted is not None else args.records)
    return {"policy": "pure dead-letter queue" if policy == "dlq" else f"halt above {args.threshold:.0%} of last {args.window:,}",
            "dead_lettered": dlq, "good_processed": good, "halted_at": halted, "records_after_change_before_stop": (halted - args.change_at) if halted else None,
            "held_back": held, "bad_after_change_in_dlq": sum(stream[args.change_at:halted if halted else args.records]) }
rows = [run("dlq"), run("threshold")]
for r in rows: print(f"{r['policy']:<30} dead-lettered {r['dead_lettered']:>6,}  good processed {r['good_processed']:>6,}  " +
                      (f"halted {r['records_after_change_before_stop']:,} records after the change ({r['held_back']:,} held back)" if r["halted_at"] else "never halted"))

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"policies": rows}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal
```
uv run dead-letter-rate-threshold.py
uv run dead-letter-rate-threshold.py --help     # every knob
```

Knobs
Knob · default	What it changes
--records100,000	Records.
--change-at50,000	Record at which the schema change lands.
--bad-after0.4	Invalid share after the change.
--threshold0.02	Halt threshold (share of the window).
--window1000	Window size.
--seed0	Random seed.

Our run

Two policies

Two policies
Policy	Dead-lettered	Good processed	Records after the change before halting	Held back
pure dead-letter queue	20.1K	79.9K	—	0
halt above 2% of last 1,000	23	50K	43	50K

Reference runApple M2 · 8 cores · 8 GB

Output · uv run dead-letter-rate-threshold.py

pure dead-letter queue         dead-lettered 20,099  good processed 79,901  never halted
halt above 2% of last 1,000    dead-lettered     23  good processed 50,020  halted 43 records after the change (49,957 held back)

macOS 26.6.2 · Python 3.13.3 · run 2026-10-20

What we found

The pure dead-letter queue set aside 20.1K records and never stopped; the output went on growing with about four in ten records missing. The threshold policy halted 43 records after the change, faster than the expected 1,000 because 40% invalid crosses 2% of the window almost at once, holding back 50K records until someone fixes the schema.

Read it

A dead-letter queue answers "what do I do with a bad record"; a threshold answers "is something wrong with the stream". Both are needed. The threshold trades availability for correctness: the halted stream is visibly late, which is the point, because the alternative is a stream that is on time and wrong.