Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Transactional outbox

How does the outbox move a second write inside one system's boundary, and who pays for the rest?

The dual-write problem: update a database and publish a message, atomically. There are two systems, so by "atomicity over n items is atomicity over one item" you would need one atomic event covering both, and there is none. If the code does UPDATE then publish, a crash between them leaves a seat reserved and no message. If it publishes first, a crash leaves a message about a state change that never happened.

The fix is structural, not clever: move the second effect inside the first system's atomic boundary. Write the message as a row in an table in the same local transaction as the state change.

BEGIN;
UPDATE seats SET status = 'sold', order_id = 981 WHERE seat_id = 'F-12';
INSERT INTO outbox (id, topic, payload)
VALUES ('evt-981-paid', 'order.paid', '{"order_id": 981}');
COMMIT;   -- one atomic event covers both the change and the intent to publish

A separate process, the relay, reads the outbox and publishes each row to the event-bus, and then marks or deletes it.

Be precise about what this accomplished. Atomicity across two systems (impossible) was decomposed into three parts:

  1. atomicity in one system (trivial);
  2. at-least-once delivery from the relay to the broker (achievable: keep trying until acknowledged);
  3. idempotent consumption (mandatory, and pushed onto someone else).

The problem was not solved. That is "costs are conserved, never removed", with the receiving party clearly named.

The costs are concrete:

  • Duplicates are guaranteed. A non-idempotent consumer is a latent bug, not a risk.
  • Per-key ordering needs the relay to be partition-aware or single-threaded per key.
  • The outbox table is a queue inside a database: rows are inserted and deleted at a high rate, so you get index bloat and pressure, which is a reliable way to recreate the problem of Multiversion and optimistic schemes. That is why mature implementations read the outbox through CDC (: reading the database's own log of changes) instead of polling the table.

The variant worth naming: skip the outbox and publish the WAL directly. The atomic event becomes the database itself, and the table disappears. The price: the published events now have the shape of your physical schema, which couples every consumer to your table layout. The outbox buys freedom in event design for the cost of a table; raw CDC saves the table for the cost of coupling. Decide deliberately, not by default.