Composition across a chain, and verification

Why does a chain have the semantics of its weakest hop, and how do you check what you really have?

Composition: semantics across a chain

This is the most common way real pipelines lose the guarantee they believe they have.

The weakest-link rule

A chain A to B to C to D has the semantics of its weakest hop. Three effectively-once hops and one at-most-once hop give at-most-once end to end. The strong hops do not compensate. They only make you confident.

Where the weak hop usually is, roughly in order of frequency:

Auto-commit somewhere in the middle.
A consumer with internal concurrency that commits the highest completed offset.
A sink that is idempotent for the main table but not for the audit log written next to it.
A DLQ that drops records without recording them.
An enrichment call whose failure is caught and logged, after which processing continues.

The last one deserves emphasis: a try/catch around a side effect, followed by continuing, is an at-most-once decision made by whoever wrote the error handler, usually without realizing it. Search your code for those. They are where guarantees quietly die.

try:
    enrich_from_partner_api(record)     # if this fails...
except Exception as e:
    log.warning("enrichment failed", e) # ...we log, and move on:
                                        # this record is now processed WITHOUT its effect.
commit(record.offset)

The end-to-end argument

This is the principle of Saltzer, Reed and Clark (a classic paper on system design), and it applies here exactly:

Per-hop exactly-once does not give end-to-end exactly-once, because the endpoints (the original source and the final sink) are outside the hops. If the source produced a duplicate (an application retry, a user double-click, an upstream replay), no transactional machinery in between can remove it.

The practical consequence: the sink must be idempotent whatever the transport promises. Every transactional feature between source and sink is an optimization of performance and convenience on top of that requirement, not a replacement for it. If your correctness argument relies on a broker feature, it is fragile. If it relies on a primary-key constraint at the sink, it is robust.

Composing across systems

Semantics do not transfer across a system boundary. Kafka's exactly-once ends at Kafka. Flink's ends at its checkpoint barrier. When output leaves, the guarantee stays behind.

Only three things cross a boundary unchanged:

An idempotency key carried in the record. It travels with the data and works everywhere.
A version or sequence number that allows conditional application.
A deterministic identity (a natural key, a content hash).

Put these into your record envelope (the standard fields around the payload) from the start. Adding an idempotency key to a schema after two years of production data is a migration. Having it from the start is one field. It is the cheapest insurance in this whole mechanism, and the one thing that survives every architecture change you will make later.

{
  "event_id": "refund:4711:r-93",
  "event_type": "OrderRefunded",
  "version": 7,
  "occurred_at": "2026-03-02T03:11:48Z",
  "payload": { "order_id": 4711, "amount": 1850 }
}

Verification

Few teams can answer "how do you know?" These checks actually work.

Reconciliation counts. Compare the records in at the source with the records out at the sink, over a window, with a tolerance and an alert. This catches loss, which nothing else does. It is the single highest-value check in this article, because loss is otherwise invisible by construction.

A duplicate-rate metric. If your sink dedups, count the rejections. A rate of zero means either you have no duplicates or your dedup is not running, and on a dashboard that shows only errors those two cases look the same. A stable non-zero rate is evidence that the machinery works.

Deliberate crash testing. Kill the consumer in the middle of a batch, again and again, and check the sink. Almost nobody does this. It finds the check-then-act race, the auto-commit surprise and the non-idempotent audit write within an hour. This is not benchmarking. It is the only way to observe the failure modes of this article, because they appear only at crash boundaries.

Replay a known window into a scratch sink and compare. This checks determinism, checks idempotency, and checks that your replay path exists and works. Do it before you need it.

For every effect, ask: what is the key and what is the store? If nobody can answer in one sentence, that effect is at-least-once with no dedup, which means it produces duplicates. This is a code-review question, not a monitoring question, and it catches problems before deployment.