At-most-once, at-least-once and the exactly-once category error
Why is exactly-once delivery impossible, and what does 'exactly-once' actually guarantee?
At-most-once, at-least-once, and the exactly-once category error
After an ambiguous timeout the sender has one bit of freedom: retry or do not retry. That is the entire space.
- Do not retry → at-most-once. No duplicates. Loss is possible.
- Retry → at-least-once. No loss. Duplicates are possible.
The name at-least-once hides a precondition: the sender must hold the message durably until it is acknowledged. If the sender crashes while holding the message only in memory, the message is lost, and you have neither guarantee. So at-least-once is not free. It is bought with sender-side durability (a log or a database row).
There is no third setting of that bit. Exactly-once delivery is not a stronger point on the same dial. It is a category error. The impossibility comes from the two generals result. It is not an engineering gap that a better protocol will close.
What is achievable is exactly-once effect: the message is delivered many times, and the observable state changes once. The receiver, unlike the channel, can hold state, and state is what the channel lacks. So the guarantee is moved from the transport to the endpoint. This is the end-to-end argument (a correctness property can be provided only by the two endpoints of an operation; lower layers can make it cheaper but cannot supply it). It is the most important cross-cutting idea in this article, and a hypothesis rather than a law. What would disprove it: a guarantee genuinely supplied by an intermediate layer, so that the endpoints need no check of their own. Schema contracts and validation gates make a claim of exactly that kind, so they are the place to look.
Three corrections to carry out of this section:
- "Exactly-once" means at-least-once delivery plus effectively-once state. Nothing is processed once. Work is repeated and the repetition is discarded. An engine that claims otherwise is describing its observable output. The CPU cost of the discarded attempts is real.
- Exactly-once has a boundary: the set of effects the system owns. An HTTP call, an email, or a write to an untracked store, made from inside a task, is outside every checkpoint and every transaction. No engine feature can include it. "End-to-end exactly-once" is a property of a closed system with a transactional or idempotent sink. It is never a property of arbitrary user code. The classic demonstration: a job that is truly exactly-once in its output table and sends duplicate notifications from inside a
mapfunction. - The word means different things in different products. In a stream processor it means state and sink advance atomically with the input position. In a message broker it means duplicate suppression and transactions inside that broker. The second does not give you the first. A producer with idempotence enabled, writing to a broker, still produces duplicates in your database if your consumer commits offsets separately.
Effects outside the boundary
Every effect your consumer produces belongs to one of four classes.
| Class | Examples | Repeatable? |
|---|---|---|
| Idempotent internal | upsert by primary key, PUT to object store, overwrite a partition | Yes, for free |
| Non-idempotent internal, dedupable | insert, increment, append | Yes, with a dedup key and a store |
| External, dedupable | payment API, partner API that accepts idempotency keys | Yes, if they support it |
| External, non-dedupable | send an email, ring a phone, fire a missile, print a check | No |
The last row is the true boundary of the whole field. For a really non-idempotent external effect, exactly-once is impossible and no architecture changes that. You must choose:
- At-least-once: possibly two emails. Usually the right choice: annoying, not harmful.
- At-most-once: possibly no email. Right when a duplicate is worse than an absence (a duplicate payment, a duplicate physical shipment).
That is the entire decision. It must be made explicitly, per effect, by someone who understands the business consequence. It is not a technical decision, and it must never be a default.
Idempotency does not compose by itself. A consumer that writes idempotently to a database and sends an email is idempotent in one effect and not in the other.
What is left can be shrunk. A partner API that accepts an idempotency key turns an external effect into a dedupable one, if the key comes from the record (Idempotency: the three sources). For the rest:
Interpose an idempotent intermediary. Instead of the consumer sending emails directly, let it upsert a row into a pending_notifications table keyed by record ID. A separate sender reads the table and marks rows as sent. You have not achieved exactly-once. You have shrunk the non-idempotent window to one small, observable, retryable component whose failures you can see and reason about. This is usually the best available outcome and the right architectural move.
Order the effects by reversibility. If you must do several things and cannot make them atomic, do the reversible or idempotent ones first and the irreversible one last. Then a crash in the middle leaves recoverable state, not an irreversible effect with no record of it.
Recording the intent before acting turns an unknown into a known unknown: Write-ahead intent and the recovery scan.