Make a repeat harmless
Retries and recovery redo work, so a write, a message or a call may land twice: an event counted twice, a row inserted twice. Something must remember what was already done, and there are only a few places that memory can live. Where should it live for this effect?
You will end with where the memory of 'already done' lives, what it costs, and the assumption it rests on.
The questions go from cheapest to most expensive answer: stop at the first one that fits.
Decide
Does a second copy of the effect change nothing, or can the effect be written to a fixed address?
SET status = 'paid', MAX, a delete; or overwriting the output file for 2026-10-21, named by the work, not the attempt.
Why
An operation that absorbs repeats, or output named by the work rather than by the attempt, makes a repeat harmless with no extra state at all.
Can the record of progress (the input offset) live in the same store as the effect?
A consumer writing to PostgreSQL can store its offset in a PostgreSQL table.
Why
If progress and effect commit together, they cannot disagree after a crash: the second write disappears entirely. This is the cheapest correct answer and the one most often skipped. With progress and effect in separate stores, no ordering of the two commits is correct.
Can the sink hold a write invisibly, then make it visible on command?
A database transaction that can be prepared, or a file staged and later renamed.
Does each producer have a stable identity and send in order?
Why
With ordered producers, the receiver needs only the last sequence number per producer, and it also detects gaps. Without them, it must remember every key it has seen, for as long as a duplicate might arrive.
Where you can land
Natural idempotence
Use it when the operation absorbs repeats, or output can be named by the work.
What it costs: only absolute operations qualify; relative ones (add 10) need a compare-and-swap. The unit of work must have a stable name.
One transaction for effect and position
Use it when the offset can be stored where the effect is written.
What it costs: couples your progress to the sink's schema, and works for one sink only.
Two-phase sink commit
Use it when the sink can hold a write invisibly until told to publish it.
What it costs: output becomes visible only at each commit, so latency is the commit interval.
Before you build it
Check the sink's timeout against recovery
The sink aborts a prepared write after a timeout. It must be longer than commit interval + failure detection + restart + backlog drain: 60 + 30 + 120 + 300 = 510 s in one worked example. A default of 60 or 300 seconds silently destroys data in the first real incident.
Sequence numbers per producer
Use it when producers have stable identities and send in order.
What it costs: state per producer, not per message, and gap detection for free. Autoscaling, which creates and destroys producers, breaks the identity it relies on.
Idempotency keys with a window
Use it when nothing cheaper fits.
Store each key with its outcome; skip repeats.
Before you build it
State the window's assumption out loud
Keys are kept for a window W, which is a bet that no duplicate arrives later than W. The bet fails exactly when the system is slowest. Size it: 100,000 keys a second at 32 bytes each is about 11 GB per hour of window.
Whichever you land on
Classify errors, and decide what a permanent one does
At the point of failure, classify errors as transient (retry) or permanent (retrying will not help). Decide in advance whether a permanent error stops the pipeline (correct, but a human is on the critical path) or skips the record (the pipeline keeps moving, with a gap downstream totals will never mention).