Make effects repeatable
A consumer reads messages and does things: writes rows, calls APIs, sends emails. Messages can be delivered more than once, and the consumer can crash between doing the work and recording that it did. Delivering exactly once is impossible, but processing effectively once is not: every effect must be harmless to repeat. How do you get there?
You will end with for every effect, how a repeat is made harmless, plus a commit policy, a replay check and the metrics that prove it.
Work through these in order.
Work through it
List every effect
Not the consumer: its effects. Include the audit log, metrics, cache invalidations and notifications, not just the main write.
Classify each effect
- Repeatable for free: upsert by primary key, overwrite a file or partition.
- Repeatable with a deduplication key: insert, increment, append.
- External, repeatable if they support it: a payment or partner API that accepts idempotency keys.
- External, not repeatable: send an email, ring a phone, print a cheque.
Decide the unrepeatable effects with the business
For each effect in the last class, exactly-once is impossible. Choose, per effect and with the business owner: at least once (maybe two emails: usually right) or at most once (maybe no email: right when a duplicate is worse than nothing, such as a payment or a shipment). Then try to make it repeatable by putting an idempotent service in front of it.
Use at-least-once delivery with an idempotent sink for the rest
Prefer effects that are naturally repeatable (upsert, overwrite, atomic replace) over a deduplication store.
If you need a deduplication store, pin it down
Name the key, the store and the retention window. Confirm the dedup record is written atomically with the effect, and that the check is a conditional write (insert-if-absent), not check-then-act.
Commit progress after processing
Turn off auto-commit: it commits offsets on a timer, possibly before the work is done, which silently makes the pipeline at-most-once. Size the commit interval by how much reprocessing the sink can absorb, because the interval is the size of the duplicate burst after a crash.
Checkpoint state and position together
If the consumer keeps state (running totals, windows): save the state and the input position together, atomically. Choose aligned checkpoints (pause inputs until everything is consistent) or unaligned ones (save in-flight data too) by whether your problem is checkpoints stalling under load or growing too large.
Check that you can replay
All five must hold: input retained long enough; positions you can start from; deterministic transforms (audit for now(), random numbers and lookups into tables that change); state that can be reset; and a sink that tolerates the replay.
Find the weakest hop
Trace the chain from source to final effect. The pipeline is only as strong as its weakest hop; fix that one first, because the others do not matter until you do.
Measure it, then break it
Add reconciliation counts (records in against records out) and a duplicate-rate metric. Then crash the consumer on purpose and watch what happens.