Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Log-based CDC

Why is reading the database log not free for the source, and how can a consumer take the source down?

The idea

The source already maintains a durable, totally ordered record of every state transition. It must, because crash recovery is impossible otherwise. So the log contains, by construction, exactly what wants: every mutation, in order, with transaction boundaries. And because the database must write the log before it acknowledges a commit, reading the log adds nothing to the write path.

Decoding

Therefore the decode step moves outside the transaction: a separate process the log (reads new records as they appear), translates physical records into logical row changes ("row 17 in orders changed status from packed to shipped"), and ships them.

This is why every mature database grows a facility separate from its physical log. The physical log is addressed in pages and byte . It is meaningless to anyone who is not the same version of the same engine. Logical decoding requires the source to log more than recovery needs: full (the old row), or at least full key images, so that a change can be interpreted without access to the page. (In PostgreSQL, for example, a table setting called REPLICA IDENTITY FULL makes the log carry the full old row; by default only the old key is logged.)

The claim "zero impact on the source" is false

Log-based CDC is routinely described as having no impact on the source. That is wrong as stated. The accurate claim is narrower and still important: log-based CDC stays off the write transaction's critical path.

What it really costs:

  • Higher write volume. Full row images often raise log volume by 10–30%.
  • CPU for decoding. The decoder competes with the source for resources.
  • Constrained retention. The source can no longer decide on its own when to delete old log files.
  • Inverted blast radius. This is the most important point. The source can no longer recycle log segments that the consumer has not read. A CDC reader that dies on Friday fills the primary's disk by Monday.

At that moment the primary database stops accepting writes, and the whole application is down. This is the clearest case in the stack of a downstream analytical consumer taking down an upstream transactional system (a long-running read query that blocks cleanup is a milder one), and it is the most under-taught fact about CDC.

This is consumer liveness as a source-side risk. The blast radius points backwards: the downstream consumer's health becomes the upstream system's availability problem. Mitigations all amount to making that risk visible and bounded: alert on the age of the oldest unconsumed log position, cap how much log the source may retain for a consumer (accepting that the consumer must then re-bootstrap if it falls too far behind), and monitor the reader as a production service.

Ownership: the principle behind the problem

The deeper issue is ownership. The log was a private implementation detail. Once you read it, its format, its retention policy and its upgrade cadence become a contract that nobody on the source side agreed to sign.

You will meet this again in schema inference, in undocumented internal tables, and in every "we just read their database directly" integration. It is the first genuinely organizational principle across the platform: it is about who bears the cost of a change, not about physics.

In one line: , because it makes your liveness the source's problem.