Dead-letter queues

What does a DLQ really trade, and how do you keep it from becoming an outage amplifier?

A dead-letter queue (DLQ) is where records go when they cannot be processed after N attempts. Whether to stop, skip or set such a record aside, and what each exit costs, is explained in Poison pills and skip-and-log. This page is about the DLQ as a route: what it does to order, and how to keep it from amplifying an outage.

What it does to your architecture

This is the point most explanations miss:

You kept the pipeline moving. To do so, you took a record out of its ordered position and put it back, if at all, hours later, out of order, perhaps after records that logically depended on it. For a log-based transport, the DLQ is the place where you locally turn your log back into a queue. That can be a good trade, but it must be a conscious one. Downstream logic that assumes per-key order is now wrong for exactly the records that were hardest to handle.

Classify the failure first

This is the most important step. It decides whether the DLQ helps or hurts.

(Backoff means waiting longer between each retry, for example 1 s, 2 s, 4 s. A circuit breaker stops calls to a failing dependency.)

The catastrophic case is the last row. The downstream system is down, every record fails, and your DLQ policy dutifully moves your entire stream into the DLQ over the course of an hour. Now you have a drained pipeline, an enormous DLQ, and a replay job that must preserve an order you have already destroyed.

Guard against it. If the DLQ rate exceeds a small fraction of the throughput, that is not a poison record. That is an outage. Stop consuming and alert. A DLQ with no rate limit is an outage amplifier.

# Pseudo-code: a DLQ with a stop condition
recent = deque(maxlen=1000)           # 1 = sent to DLQ, 0 = processed ok

def handle(record):
    try:
        process(record)
        recent.append(0)
    except TransientError:
        retry_later(record)           # never DLQ a transient failure
    except PermanentDataError as e:
        send_to_dlq(record, error=e)
        recent.append(1)
        if sum(recent) > 20:          # more than 2% of the last 1000 -> this is an outage
            pause_consumption_and_alert()

(The 2% threshold is only an example. Choose a threshold from your own normal failure rate.)

Design requirements

• Retry topics with escalating delay, not a single DLQ: retry-5s → retry-1m → retry-30m → DLQ. This separates transient from permanent failures without blocking the main stream, and the escalation itself is diagnostic.
• Preserve the full context: the original payload, the original topic, partition and offset, the error, the stack trace, the attempt count, the time of the first failure and the consumer version. A DLQ record without provenance cannot be replayed, and a DLQ you cannot replay from is a very expensive /dev/null (a file that throws away everything written to it).
• Never replace the original payload with an error message. This happens more often than you would think.
• A replay path that exists and has been tested. An untested replay is not a recovery plan.
• Monitor depth and age. Depth catches bursts. Age catches the DLQ that quietly collected 40,000 records over six months because nobody owned it. An unmonitored DLQ is data loss with extra steps, and this is the most common outcome of adopting a DLQ.
• Idempotent consumers, because replay means reprocessing, and some replayed records were partly applied before they failed.

{
  "original_payload": "{\"order_id\":\"o-4711\",\"type\":\"OrderPlaced\", ...}",
  "source": {"topic": "orders", "partition": 7, "offset": 1893021},
  "error": "SchemaViolation: field 'total' is not a number",
  "attempts": 3,
  "first_failed_at": "2026-11-27T10:02:11Z",
  "consumer_version": "inventory-consumer 2.14.1"
}

Ordering on replay

There are two honest options. Replay to the original topic: downstream processing is correct, but the record lands at an arbitrary position in the stream. Replay to a side path with special handling: the main stream is preserved, but the downstream system must accept out-of-band corrections. Neither restores the original order. If order for those records is truly required, your only real option was blocking, which means the DLQ was the wrong mechanism and you should have paused the partition instead.