Transactional sinks and two-phase sink commit
What must a sink support for exactly-once output, and which timeout silently voids the guarantee?
A stream or batch engine keeps its position and state in its checkpoint. Its output goes to a sink, a foreign system (a database, a topic, a file store). That is two stores with no shared transaction, and a guarantee that requires them to agree. So the engine becomes a two-phase commit (2PC) coordinator and the sink becomes a participant. (2PC: phase one asks every participant to prepare, meaning to make the change durable but not yet visible; phase two tells them all to commit, or all to abort.) The protocol itself is standard. What matters here is the interface requirement it places on the sink, because that decides whether exactly-once is available to you at all.
The sink must support a durable, invisible, later-committable write. This is a strong and unusual property.
| Sink | Has it? | How |
|---|---|---|
| Filesystem | Yes | write to a temp file, then rename |
| Kafka-style log | Yes | transactions and read_committed readers |
| Relational database | Yes | native transactions |
| Most HTTP APIs | No | |
| Most key-value stores | No |
When the sink lacks this property, no engine configuration produces exactly-once. The honest alternative is the idempotent sink: deterministic keys or paths, overwrite semantics, no prepare phase, no blocking, no coordinator. That path is cheaper and more robust. Its only requirement is that the unit of work has a stable address, which is Source 2 of the idempotency section promoted to an architecture.
There are three costs. One of them is the most common real-world break of this guarantee.
Cost 1: latency is quantized to the commit interval
Output is invisible until the epoch commits. So a transactional sink turns a record-at-a-time job into a micro-batch job from the consumer's point of view. The guarantee is bought with acknowledgment latency. The checkpoint interval becomes a product decision, not an operations setting. If the interval is 10 seconds, a downstream reader sees results in steps up to 10 seconds late.
Cost 2: the prepared transaction has a timeout, and it is usually shorter than your recovery
A prepared transaction cannot wait forever, because it holds locks and resources. The sink aborts it after a timeout. If the sink aborts a prepared transaction before the engine comes back and commits it, the guarantee is void and the data is gone. The inequality that must hold is:
This is the point most likely to be misconfigured in a real deployment. The default timeouts are tuned for normal operation, and the constraint binds only during failure, when the right side of the inequality is at its largest. The incident that made the transaction pending is the same incident that makes recovery slow.
Cost 3: the commit itself is ambiguous
The coordinator sends "commit" and loses the reply. Did the commit happen? The regress (the endless chain of "and did that acknowledgment arrive?") stops only because the commit operation is made idempotent: committing again is a no-op. This is the section's own medicine applied at its own last step. It is also a clean example of the principle "the recovery path should be the normal path": the code that re-commits after a crash is the same code that commits normally.