Poison pills and skip-and-log
What do you do with a message that can never be processed, and what does a dead-letter queue hide?
Everything so far assumes that retry eventually succeeds. A poison pill is a message that can never be processed, for example a malformed record. Its failure is deterministic: this is the permanent error class from Why recovery is different. Meeting an at-least-once mechanism, it produces an infinite loop. With ordered delivery, it also blocks every message behind it in the partition. This is head-of-line blocking (one stuck item blocks the line) at the level of meaning instead of the network.
There are three exits. They are the three ways every design in fault tolerance pays: by waiting, by wasted work, or by weakened semantics.
| Exit | What is paid | Currency |
|---|---|---|
| Stop (stop the line) | availability is sacrificed | waiting |
| Skip (drop and move on) | semantics are weakened: silent loss | weakening |
| Quarantine (set aside for later) | work is deferred; an owner is required | wasted work |
This is the only mechanism in the part where the right answer is genuinely a business decision, not something you can derive. A financial ledger stops. A clickstream skips.
Three honest observations about the dead-letter queue
A dead-letter queue (DLQ) is a side queue where failed messages are put. It is the default answer and the weakest one.
- A DLQ silently breaks completeness. Downstream aggregates are now computed over a subset, and nothing in the data says so. A DLQ is correct only if the consumer's meaning tolerates gaps, or if something reconciles counts between source and sink. Without one of these, "skip-and-log" is data loss with a paper trail nobody reads. The useful discipline: treat DLQ depth as a data-quality signal with an owner and a service-level target, not as an error dump.
- Skipping breaks order for that key. If the poison record was an update to an entity and you skipped it, later updates apply to a state that never existed. So DLQ-and-continue is safe for independent records and unsafe for a changelog (a stream of changes that must be applied in order). Most implementations do not make this distinction.
- The retry counter must travel with the record. If it is held in process memory, it resets on restart. A poison pill that kills the consumer then produces infinite retries across infinite restarts, and the "give up after N" rule never fires. Keep the counter in message headers or in a keyed side store.
Stronger delivery guarantees do not remove a poison pill either. Under exactly-once processing with transactions, the poison record's transaction aborts and replays: the crash loop becomes an infinite abort-and-retry loop that consumes capacity and makes no progress. The guarantee changes the shape of the stall, not its existence.
The classifier is asymmetric
A record that fails because a downstream dependency is down looks exactly like a poison pill to a rule that counts attempts. A naive threshold therefore quarantines good data during an incident, at peak volume. It is the same shape as node blacklisting in Recovery mechanisms, and the mitigation is the same: separate the error classes at the source (a parse error is permanent; a refused connection is not), instead of guessing the class from the attempt count.