Where loss, duplicates and reordering come from

At a crossing, where do loss, duplicates and reordering come from, and which of them should you choose?

CaseConstructed case · Lena at Cartwheel

A replay doubled last month's revenue

Lena finds a bug in the revenue job: for three weeks it applied the wrong currency rate to orders from two sellers. She fixes the code and replays those three weeks of OrderPlaced events from the log into the revenue table. The next morning, revenue for most of the replayed days is exactly double, and days the bug never touched have moved by a few cents.

Two weeks later the finance team brings a second puzzle. Kafka's exactly-once settings are on and every stage reports success, yet the warehouse holds 0.6% fewer orders than the Orders service produced, and no error was logged anywhere.

Omar asks: "We replayed to fix a number and made it worse, and we lose orders with nothing in the logs. What do our guarantees actually cover?"

How it ends →

Delivery semantics: the basics showed that the order of two writes, the acknowledgement and the effect, chooses between loss and duplication at one crossing. This article follows the question across a whole pipeline: where loss, duplicates and reordering come from, how a replay re-derives past output, and how a chain of crossings keeps or loses its guarantee. Making a repeat harmless (idempotency keys, offsets stored with the effect, transactional sinks, effects outside the boundary) is the subject of Fault tolerance; this article links there instead of repeating it.

Delivery is not processing

Delivery semantics describe the transport: how many times the bytes reached the consumer. Processing semantics describe the effect: how many times the consequence happened in the world. A transport can give you only the first. The target is at-least-once delivery and effectively-once processing, and the second is built in your consumer and your sink (why exactly-once is about effects).

Where each one comes from

Each failure has a short list of causes at a crossing, and each cause is explained where it lives:

Outcome	Comes from	Explained in
Loss	progress recorded before the effect (ack first, or auto-commit); an in-memory buffer lost in a crash; a record skipped without being written anywhere; a consumer away longer than retention; a `try/catch` that logs and moves on	the basics, auto-commit, Buffering, Dead-letter queues, publish/subscribe, composition
Duplicates	a crash between the effect and the commit; redelivery after a visibility timeout while the first consumer is still working; a rebalance that moves a partition before its offset is committed; a producer retry after a lost acknowledgement; a replay	the basics, Queue versus log, rebalancing, Epochs and sequence numbers, Replay
Reordering	retries with several requests in flight; parallel work inside one consumer; DLQ re-injection; a change in the partition count; fan-in without a watermark	Message keys and key-based ordering, Dead-letter queues, the shapes

Duplicates and reordering can be repaired downstream, because the record is still there. Loss cannot, which is why the choice below is lopsided.

Why at-least-once is almost always right. A duplicate can be detected and corrected: you hold the record, so you can compare it with state you control. Loss can be neither. A lost record leaves no evidence anywhere. You cannot write a check for it, because the check would be "count the records that do not exist".

The narrow exception. At-most-once is a legitimate engineering choice when the value of a record falls to zero faster than a retry could happen, and the volume is so high that duplication costs more than loss. Examples: high-frequency metrics, sensor telemetry, sampled traces, presence updates ("user is online"). In those cases it is a decision. Everywhere else it is an accident.