Change event envelopes and retraction
Why does a change event that lacks the old values break computation downstream?
What an event must carry
A change event must carry enough to be applied, applied idempotently, and applied in order, by a consumer that has lost all its local context. That forces four things (and derives the rest). The structure that holds them is called an envelope.
| Field | What it does |
|---|---|
| Op code (insert, update, delete) and after-image (the new row) | lets you apply the change |
| Before-image (the old row) | lets you retract (undo) the old version. Its presence or absence decides which computations the consumer can maintain |
| Source position (LSN, SCN or GTID) | is both the idempotence key and the ordering key. The consumer's deduplication and its cursor are the same value |
| Transaction ID and commit position | lets you reassemble atomic groups of changes |
| Source timestamp and capture timestamp | two different clocks (when the change happened, and when your reader saw it). Their difference is the only honest freshness metric you have |
An example event:
{
"op": "u",
"source": { "table": "orders", "lsn": 8100234, "txid": 5521, "ts_ms": 1767225600000 },
"before": { "order_id": 17, "status": "packed", "total": 40 },
"after": { "order_id": 17, "status": "shipped", "total": 40 },
"captured_ts_ms": 1767225600420
}Here the freshness lag is captured_ts_ms − source.ts_ms = 420 ms.
Signed multiplicity: the unifying idea
The unifying abstraction is not "insert / update / delete". It is signed multiplicity: every change is a row with a count.
- an insert is
(row, +1); - a delete is
(row, −1); - an update is exactly two entries:
(old, −1)and(new, +1).
Every downstream question then becomes: "can this operator absorb signed changes?"
This is why a missing before-image is not a minor omission. Without it, an update cannot be written as (old, −1), (new, +1), so it cannot be expressed as a retraction. The consumer is then restricted to state-convergent operations only (such as "set the row to this value"). It can no longer maintain a sum, a count or a join incrementally.
In one line: an envelope is idempotent and ordered if it carries its own position; it is composable with computation only if it carries what it replaced.