Idempotency: the three sources

Why is every dedup window a bet?

An operation is idempotent when applying it n ≥ 1 times leaves the same observable state as applying it once. Read this definition carefully. It says nothing about the response. A second PUT may return 200 where the first returned 201. A second DELETE may return 404. If the caller treats a different response as a different outcome, then the effect was idempotent and the system still is not.

There are exactly three sources of idempotence. They cost very different amounts.

Source 1: natural (algebraic) idempotence

Some operations are already absorbing: applying them again changes nothing. Examples: SET x = 5, MAX, INSERT IF NOT EXISTS, delete-by-key, writing to a deterministic path. This is free. The right algebra removes the need to coordinate.

The boundary is sharp, and it is the whole game: absolute operations are idempotent, relative ones are not.

You can convert a relative operation into an absolute one: the writer reads the old balance (100) and writes the new value (110). But now another writer may change the balance between the read and the write, and you have a lost-update problem (two writers read 100; both write 110; one +10 disappears). The usual cure is compare-and-set (CAS): "set the balance to 110 only if it is still 100". So the escape is real, but it follows the "costs are conserved" principle exactly: you move the cost from the retry path to the concurrency path. Compare-and-set is useful because one primitive solves both problems.

Source 2: positional (address-derived) idempotence

The retried unit of work writes to an address that is a function of the logical work. Then overwrite absorbs the duplicate. This is how batch systems get idempotence almost for free: overwrite the partition for the logical date, use a deterministic output path, replace rather than append.

The failure mode is precise and very common. The address is derived from the attempt instead of the intent: a run id, now(), an attempt number appended to a file name. The retry then lands beside the original instead of on top of it.

BAD : /out/billing/run-8841-attempt-2/part-0007.parquet     (named by when it ran)
GOOD: /out/billing/date=2026-03-14/part-0007.parquet        (named by what the work is)

This is the same rule as an idempotency key, one layer down. A pipeline that appends instead of overwriting has no idempotence at all, whatever the job scheduler claims.

Source 3: imposed idempotence (a dedup record)

When neither of the above is available, you must remember. That costs state, and the state is where the guarantee actually fails. The standard form is the idempotency key: a unique name for one intended operation, sent with the request. The receiver keeps a table of keys it has seen.

Rules that production systems often get wrong:

1. The key must be generated by the client, before the first attempt, and reused across attempts. If the server generates the key, the retry is a new request with a new key, and the mechanism is decoration. The key is a name for the intent.
2. The endpoint must store key → outcome and replay the stored outcome. If it only suppresses the write, a retry gets a "uniqueness violation" error. The caller then cannot tell "I already did it" from "someone else did it". The ambiguity is not removed; it moves into an error code.
3. The dedup record must be committed in the same transaction as the effect. If they are in different stores, you have recreated the original problem one level down. This is exactly the situation that makes the outbox pattern necessary (the outbox pattern writes the message to be sent into the same database transaction as the business change, and a separate relay sends it later).

Two more rules about the key itself:

• Scoped to the operation. A key that is unique per record but not per operation fails when one record causes two effects. Use {record_id}:{operation} where needed, for example 4711:refund and 4711:restock.
• Content-hash caveat. Two legitimately identical records (the same user clicking twice, two identical sensor readings) look exactly like a duplicate. A content hash mixes up "same content" with "same event". If legitimate repeats are possible, include a sequence number or a timestamp.

A small example in SQL, for Mediflow's own database:

-- key = the lab result id (the intent), never a random id per attempt
BEGIN;
INSERT INTO processed_results (result_id, outcome)
VALUES ('LR-90417', NULL)
ON CONFLICT (result_id) DO NOTHING;
-- if 0 rows were inserted, this result was already handled:
--   SELECT outcome FROM processed_results WHERE result_id = 'LR-90417';
--   and return that stored outcome to the caller.
INSERT INTO patient_records (result_id, patient_id, payload) VALUES (...);
UPDATE processed_results SET outcome = 'stored' WHERE result_id = 'LR-90417';
COMMIT;

Dedup windows and the bet nobody states

Exact deduplication requires remembering every key forever: unbounded state against unbounded time. So everyone keeps a window W (for example, the last hour of keys). The correctness condition is

and the maximum duplicate delay is a quantity that is unbounded in an asynchronous network (a network with no guaranteed upper limit on message delay). Every dedup window in production is a bet that the retry arrives before you forget the key. And the bet fails exactly when the system is slow, which is when retries happen. This is the independence assumption turned against you: the failure that creates duplicates also lengthens their delay.

Size the window against the largest realistic gap between the original and the duplicate:

The last row breaks the usual assumption. It is why a 24-hour dedup window and a "replay last month" capability cannot coexist, and why replay usually needs a different mechanism from online dedup (Replay).

This is why windows in the field are measured in minutes to hours. It also explains the temptation to use approximate structures such as Bloom filters (a compact structure that answers "have I seen this?" with possible false "yes" answers, never false "no" answers). That brings a trap:

A false positive says "seen before" about something new. The new record is then dropped. In most pipelines this is the wrong direction: silent loss instead of a survivable duplicate. Everywhere else, an approximate structure errs toward doing more work (an unnecessary read, a spurious conflict). Here it errs toward doing less. Check which way an approximation's error points before you adopt it. A Bloom filter is safe only as a pre-filter in front of an exact check: "definitely not seen" can skip the exact lookup, and "maybe seen" goes to the exact table.

Two consumers, one key

Two consumers process the same record at the same time. This happens during a rebalance (the group reassigns partitions) or when a record is redelivered while the first consumer is still working. Both look in the dedup store, both see nothing, both proceed.

The fix is one atomic conditional operation: INSERT ... ON CONFLICT, Redis SETNX ("set if not exists"), a compare-and-swap, or a unique constraint that makes the second write fail. Let the database enforce it. Do not enforce it in application logic.

A replaced consumer that has not noticed yet (a zombie) is the same race stretched over time; an epoch that the sink checks closes it (Epochs and sequence numbers).