Epochs and sequence numbers: one number instead of a dedup set
How do an epoch pointer and a per-producer counter replace a set of seen keys, and what does each require?
Epoch-based output
Per-record dedup keeps state proportional to the number of distinct keys, for a long time. Epoch-based output keeps one number. That difference is the whole motivation. The method: divide the world into epochs (batches of work with a number), write the output eagerly, and make visibility a function of a single designated commit. An unbounded dedup problem becomes a bounded visibility problem. Coarsening the atomicity unit from "record" to "batch" shrinks the bookkeeping by the batch size.
The mechanics are the same object you meet in many places: the WAL commit record, the manifest swap, the Iceberg snapshot pointer (a table format whose current snapshot is one pointer to a list of files), write-audit-publish (write data to a staging area, check it, then publish it by one switch), epoch-based reclamation in memory management, and terms in Raft (a consensus protocol). One pointer move decides whether n earlier writes count.
Two properties make epochs preferred over per-record dedup in distributed engines.
It fences zombies for free. A zombie is an executor that was declared dead, kept running, and produced output tagged with an old epoch. That epoch will never be published, so the zombie's output never becomes visible. The mechanism that removes duplicates also removes split-brain output, because both are "work from a run that did not commit". Getting two guarantees from one designated write is why engines converged on this design instead of receiver-side dedup.
Its failure granularity is the epoch. The epoch is all-or-nothing. A single failure re-executes the whole epoch. So epoch length trades latency against redone work. The cost per unit time is roughly C/I + I/(2·MTBF), where C is the cost of one commit, I is the interval, MTBF is the mean time between failures, and I/2 is the average redone work after a failure. This is the same curve as the checkpoint interval, because it is the checkpoint interval. A short epoch pays C often; a long epoch loses much work.
The boundary: the guarantee lives in the reader
Uncommitted output is usually physically present: files sit on the path, records sit in the topic. They are invisible only because readers agree to consult the pointer or to use read_committed. One consumer that reads the raw files, one directory scan, one SELECT against the staging path, and the guarantee is gone, with no error anywhere. The writer cannot enforce it. This is the end-to-end argument appearing a second time, from the opposite direction.
Deduplication on sequence numbers
If a producer has a stable identity and its messages are ordered, the dedup set collapses to one integer: the highest contiguous sequence number received. The receiver says: "I have seen producer P up to number 812. A message numbered 812 or lower is a repeat. A message numbered 813 is next." State goes from O(messages in the window) to O(producers). That is the difference between gigabytes and kilobytes, and it is what makes broker-level dedup affordable.
The precondition is per-producer ordering
If messages can be reordered, one integer is not enough. You need a window bitmap over the outstanding sequence numbers. This is exactly TCP's receive window with selective acknowledgment, reached by the same reasoning. Keep that link in mind: in-flight limits exist for the same reason in both systems. A producer that allows more in-flight requests than the dedup window allows can silently reorder and break the guarantee. So the in-flight limit is not a throughput tuning knob. It is the width of the dedup structure.
Sequence numbers also detect loss
A set of keys tells you a record is a repeat. It can never tell you a record is missing. A monotonic counter sees gaps. If the receiver has seen 811 and now receives 813, message 812 is missing. So a sequence number is the only dedup mechanism that converts silent data loss into a loud error at the receiver. This is large and under-appreciated. It is the reason to prefer ordered identifiers over opaque UUIDs as idempotency keys whenever the producer can supply them.
The identity must outlive the process
If a restarted producer gets a fresh id, its replayed messages carry new identities, and dedup fails at the exact moment it was needed. So systems use stable transactional ids, plus a producer epoch to fence the previous incarnation. This creates a real operational conflict: the id must be stable across restarts and also unique across concurrent instances. Autoscaling, or any deployment that briefly runs two instances, makes these two requirements collide. No general solution exists. The practical one is to derive the id from a stable partition assignment, not from the instance.
The window bet again
Per-producer state is kept for a finite time. An idle producer's entry expires, and its late retry is then admitted as new. The same window bet returns, one level down.