Skip to content
About
Main content

Change event envelopes and retraction

Why does a change event that lacks the old values break computation downstream?

What an event must carry

A change event must carry enough to be applied, applied idempotently, and applied in order, by a consumer that has lost all its local context. That forces four things (and derives the rest). The structure that holds them is called an envelope.

FieldWhat it does
Op code (insert, update, delete) and after-image (the new row)lets you apply the change
(the old row)lets you retract (undo) the old version. Its presence or absence decides which computations the consumer can maintain
Source position (, SCN or GTID)is both the idempotence key and the ordering key. The consumer's deduplication and its are the same value
Transaction ID and positionlets you reassemble atomic groups of changes
Source timestamp and capture timestamptwo different clocks (when the change happened, and when your reader saw it). Their difference is the only honest freshness metric you have

An example event:

{
  "op": "u",
  "source": { "table": "orders", "lsn": 8100234, "txid": 5521, "ts_ms": 1767225600000 },
  "before": { "order_id": 17, "status": "packed",  "total": 40 },
  "after":  { "order_id": 17, "status": "shipped", "total": 40 },
  "captured_ts_ms": 1767225600420
}

Here the is captured_ts_ms − source.ts_ms = 420 ms.

Signed multiplicity: the unifying idea

The unifying abstraction is not "insert / update / delete". It is signed multiplicity: every change is a row with a count.

  • an insert is (row, +1);
  • a delete is (row, −1);
  • an update is exactly two entries: (old, −1) and (new, +1).

: "can this operator absorb signed changes?"

This is why a missing before-image is not a minor omission. Without it, an update cannot be written as (old, −1), (new, +1), so it cannot be expressed as a . The consumer is then restricted to operations only (such as "set the row to this value"). It can no longer maintain a sum, a count or a join incrementally.

In one line: an envelope is and ordered if it carries its own position; it is composable with computation only if it carries what it replaced.