Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Replay and dual-write avoidance

Why is the replay horizon your retention, and why must you never write one fact to two systems?

Replay is a retention choice

Replay means re-reading old input and recomputing. It is only possible if the input still exists in its original form. So "kappa-style" reprocessing (fix the code, then the stream from the start) is not so much an architecture as a retention choice:

Replaying the last 30 days when the bug is 90 days old is not a partial fix. It is a full failure: you now have two periods with different logic and no record of where the boundary is. For Tariq's 60-day bug with 30-day retention, the older 30 days must be rebuilt from the source table (a full recompute), not from the log.

Why dual writes fail

A dual write means the application writes to the database and to a message broker in the same piece of code:

db.insert(order)                  # step 1
broker.publish("order_created")  # step 2  <- the process can crash right here

Doing both atomically requires atomicity across two systems, which means two-phase commit (2PC) (a protocol where a coordinator asks every system "can you ?" and only then tells all of them to commit). Almost nobody runs across a database and a broker, because it hurts availability and is operationally heavy. Without it, a crash between the two writes leaves the systems divergent permanently.

And the probability is not small. It is the failure rate times the size of the window, on every write. Tariq's 1-in-2,000 is typical: if crashes, timeouts or deployments hit the gap between the two writes for even a tiny share of requests, millions of orders will produce thousands of inconsistencies.

The resolutions: one source of truth

All resolutions reduce to one rule:

  1. CDC from the database. The write to the database is the event. Another process reads the log and publishes.
  2. The transactional outbox. The application writes the event into an outbox table in the same database transaction as the order. A separate process the table and publishes the events.
BEGIN;
INSERT INTO orders (order_id, customer_id, total) VALUES (9001, 51, 40);
INSERT INTO outbox (event_id, topic, payload)
VALUES (gen_random_uuid(), 'order_created', '{"order_id": 9001}');
COMMIT;   -- both rows exist, or neither does

That is the point: it makes your interface explicit instead of borrowed, which avoids the private-detail-becomes-public-interface problem. Its price is a write amplification (one more row per event) that you control.

The recurring structural requirement: commit the cursor with the state

Across all of these, one requirement keeps returning: the consumer's position in the change stream and its derived state must be committed atomically. If state commits first and the position lags, a crash reprocesses the same changes (duplicates); if position commits first and the state lags, it skips changes (loss). Why the order of the two commits is the delivery semantic, and how storing the position in the sink removes the choice, is Consumer offsets and commit ordering.

In one line: never write the same fact to two systems; write it once and derive the second, and commit your with your state, not beside it.