Where the memory of 'already done' lives
Why can a duplicate not be prevented at the sender, and where can the record of 'already done' be kept?
Where does the memory of "already done" live?
Recovery mechanisms deliberately created duplicates. Retry, replay and re-execution all repeat work, because the only way to be sure the work happened is to do it again. This article is the invoice for that choice, and the invoice has exactly one line item.
The reason it cannot be waived is the two generals result. Two generals on two hills can only talk by sending messengers, and a messenger may be captured. Whatever they agree, the last message in the exchange can be lost, and the general who sent it can never be sure it arrived. Every acknowledgment needs its own acknowledgment. In computing terms: over a channel that can lose messages, no finite protocol gives the sender knowledge that the effect occurred. The sender's ambiguity is permanent.
So a duplicate cannot be prevented at the source. It can only be absorbed at a place that holds state. That place is either the receiver or the effect itself.
This one fact reduces seven separate topics to one question:
The topics of this article are five answers to that question, plus the discipline for the failures that never succeed:
| Mechanism | Where the "already done" memory lives |
|---|---|
| Idempotency keys | one entry per request, at the receiver |
| Sequence numbers | one integer per producer |
| Consumer offsets | one integer per partition |
| Epochs | one number per batch |
| Transactional sinks | inside the sink's own transaction |
They are five placements of the same bit of information.
All five end at the same requirement: the effect and the record of the effect must become durable together. If they do not, the ambiguity only moves down one level. This is the principle "designate one write": atomicity over n things is always built from atomicity over one thing. If you need many things to happen at once, you pick one single write (a commit record, a rename, a pointer swap) whose success makes all of them count. This article applies that principle five times in a row. The mechanisms look different, and they are not.
One more thing to watch. Every mechanism here delays an acknowledgment or delays when output becomes visible. The delay is the price of the guarantee. We will point to it each time.