Delivery semantics: the basics

Where do at-most-once and at-least-once come from?

Delivery semantics is the deep content of the transport choice, and it is where the correctness of this whole mechanism lives. This page gives the basics: where loss and duplicates come from at a crossing. Making a repeat harmless is Fault tolerance's subject.

The three levels, and where they come from

They are not features you configure. They are consequences of the order of two operations: acknowledge and process.

You do	You get	On a crash in between
Ack, then process	At-most-once	the message is lost
Process, then ack	At-least-once	the message is duplicated
Process and ack atomically	Exactly-once (within one system)	neither

That is all. Every duplicate bug and every silent-loss bug you will ever debug is a question about where the ack sits relative to the side effect. When someone says "we get duplicates after a restart", the answer is already in this table. Here it is as code:

# At-least-once: process first, ack after.
for msg in consumer.poll():
    send_confirmation_email(msg)   # side effect
    # <-- crash here: email was sent, ack was not, so the message comes back
    consumer.ack(msg)

This is Ravi's third problem: the consumer crashed between sending the email and acknowledging. After the restart the broker delivered the message again, and the customer received a second email.

Exactly-once: what is real and what is not

End-to-end exactly-once across arbitrary systems is impossible. The reason is the lost-ack ambiguity: the confirmation is itself a message that can be lost, so the sender can never learn whether the receiver got the message (why exactly-once delivery is a category error).

So duplicates on the wire are unavoidable. The only real question is where you deduplicate: in an idempotent sink (the default), in a transaction inside one system, or in a sink that takes part in two-phase commit. Where the memory of "already done" lives explains each, and where its guarantee ends. The honest name for what most systems achieve is effectively-once: at-least-once delivery plus an idempotent or deduplicating consumer. For Ravi's confirmation emails, the idempotent sink looks like this:

-- Each confirmation is recorded once per order. A second attempt changes nothing.
INSERT INTO emails_sent (order_id, kind, sent_at)
VALUES (4711, 'confirmation', now())
ON CONFLICT (order_id, kind) DO NOTHING;
-- Send the email only if this INSERT affected 1 row.

(This makes the record idempotent. Sending the email and writing the row are still two steps, so a crash between them can still lose or repeat one email. That is the problem of effects outside the boundary. For email, the usual choice is to accept a rare duplicate.)

How to choose the key and the window, and what Kafka's idempotent producer does and does not deduplicate, are in Idempotency: the three sources and Epochs and sequence numbers.

Order and acknowledgement

Order exists only within a scope: a partition, a key, a session, a connection. Retries, parallel consumers and DLQ re-injection all reorder a partition; the sources, and whether you need order at all, are in Message keys and key-based ordering.

What the producer's acknowledgement promises (acks=0, 1 or all, with min.insync.replicas), and why most brokers acknowledge from the page cache, is in Durability across a pipeline.

Case, resolvedConstructed case · Ravi at Cartwheel

Records gone, a stalled partition, two emails

Moving OrderPlaced off direct calls, the team found that a task queue had deleted last month's orders before a bug fix needed them, one corrupt record stalled a Kafka partition while dashboards stayed green, and a consumer restart sent two confirmation emails. Where it started: Transport patterns →

Problem 1 (no replay). Lena had chosen a queue for facts. Acked messages are deleted, so last month's orders could not be re-read. She moves OrderPlaced to a log with 30 days of retention. Now a new consumer, or a fixed analytics job, can reset its offset and read from the start. (She sets retention knowingly: shortening it later would shorten every consumer's allowed outage.)

Problem 2 (the stalled partition). The corrupt record is a permanent defect, so blocking and retrying can never work. Lena changes the consumer: after 3 failed attempts it writes the record to an orders.rejected topic with the error, advances, and raises an alert. She notes in the runbook that this breaks per-key order for that one order, and she chooses pause that partition and alert for failures that look transient. The partition no longer stalls silently.

Problem 3 (two emails, double stock decrement). The confirmation consumer processes first and acknowledges after, so it is at-least-once. A crash between the two steps re-delivers the message. Ravi cannot make delivery exactly-once, so he makes the effect safe to repeat. For stock he stops sending deltas ("minus 1") and instead applies a full-state upsert keyed by (order_id, line_no) in the Inventory table, so a second delivery changes nothing. For email he records (order_id, 'confirmation') in a table with a unique key and accepts a very rare lost email for the one case where a crash hits exactly between the insert and the send. Facts go on the log, tasks (email, image resizing) go on a queue, and Ravi names the key and the store for each effect in one sentence: "key = order_id, store = the emails_sent and inventory tables."