A derivative from a system that did not plan to be read

What two new constraints appear when you read changes out of a system you do not own?

Everything before this mechanism processed either state (the current contents of tables) or events that were born as events (clicks, sensor readings). This mechanism is about a third thing: extracting a derivative from a system whose job was to maintain a state, and which regards its own history of changes as private bookkeeping.

Think of the write-ahead log. It was created so that one reader, the recovery process, could rebuild the database after a crash. In change data capture (CDC), the same log gets a second reader that it was never designed for. This is the whole reframing: CDC turns the database's private diary into a published interface.

Two new constraints appear that earlier topics did not have.

The first constraint is ownership. Until now we mostly worked with systems we control. CDC crosses a boundary. The source is operated by someone else (Lena). Its internal format is a vendor's implementation detail. Its schema changes without your consent. Its DBA can do things to it that leave no trace you can read. This is an organizational and informational constraint, not a physical one, and it causes most of the failures here.

The second constraint is path dependence. Full recomputation (rebuild everything from the current input) is a pure function of the current input. It is self-correcting by construction: every run throws away the mistakes of the last run. Incremental processing is a fold over a history: you start with a state and apply change after change. A fold is correct only if no element of the history is lost, duplicated or reordered. The moment you stop recomputing, correctness stops being a property of today's run. It becomes a property of every run that ever happened.

What stops mattering: the total size of the dataset. What starts mattering: the completeness and order of the change history, and your ability to prove either.

The governing question of this mechanism is therefore not "how do I capture changes?" but:

And the bargain in one sentence:

A few neighbors are worth separating so that the scope is clear. The write-ahead log as a durability device (it makes crashes survivable) belongs to durability. Here we study the log as a published interface. Deciding which clock a record follows (event time or processing time) belongs to time semantics; we assume that and ask how a change is detected and applied. Continuous computation over a stream that already exists is also elsewhere; we ask where the stream comes from when the source is a mutable table, and what happens when a delta lands on derived state. Exactly-once machinery is a separate topic; we are a consumer of it. Compaction and vacuum (background cleanup of old file versions) belong to table maintenance; here we meet them as the process that deletes the change history we want to read. Slowly changing dimensions (SCD, history-keeping tables) are a modeling choice; here we study the plumbing that feeds them.