Schema change propagation

How do schema changes travel in the stream, and which ones have no clean solution?

DDL is a change in the shape of all later changes, and it arrives in the same stream as the data it reshapes. So it must be ordered against the data: events before the DDL follow the old shape, events after follow the new shape. This means a schema is a versioned, in-band fact, not a static configuration file.

This is why systems keep a schema registry (a service that stores each schema version and gives it an id) and attach a fingerprint (a short id of the schema version) to every event. The reader must resolve each record against the writer's schema as of that record.

Which changes old readers survive is the question of compatibility: adding a nullable column passes, dropping or narrowing a column does not. So CDC pipelines survive additive evolution automatically, and require a coordinated migration for anything else. And the source's DBA, who owns neither your pipeline nor your calendar, decides which happens.

Two cases with no clean solution

A type change in the middle of the stream. The same column now has two types inside one topic, and no single sink schema is correct for both. (This is Lena's integer-to-decimal change.)
A DDL that rewrites the table physically. It may produce nothing in the log (a metadata-only operation you never see), or it may rewrite every row, producing an update storm of the whole table that cannot be told apart from real activity.

ALTER TABLE orders ADD COLUMN note text;                 -- metadata only: usually nothing in the log
ALTER TABLE orders ALTER COLUMN total TYPE numeric(12,2); -- may rewrite every row: a storm of 300M events

Both are correct behavior of the source. Both are catastrophic for you. This is the private-detail-becomes-public-interface principle again: you are consuming an interface that nobody promised to hold still.

In one line: schema is an in-band, ordered event; additive changes propagate themselves, and everything else needs a human on both sides.

Case, resolvedConstructed case · Tariq at Quillpost

4,000 orders with an old status after the copy

Copying 300 million existing rows while the stream ran left stale and reappearing rows, a refund between two wallets briefly showed money missing, and a column type change broke the sink. Where it started: Bootstrap and represent →

The stale and reappearing rows. His plain SELECT * was not coordinated with the stream. He switches to the incremental snapshot: chunk by key range, watermark markers in the log, and the rule "a streamed change at a higher position wins". Because his sink is a keyed upsert table, overlap is harmless. After the re-run, no stale or resurrected rows are found.

The missing money. He sees that key partitioning splits the transfer. He chooses the usual escape (accept intermediate states) and tells finance that the dashboard is only consistent at the end of each 5-minute window, never within it. For the one report that must show transfers in balance, he reads from the warehouse only up to a log position that every partition has passed (the lowest of the partitions' positions).

The type change. He pins the schema registry to compatibility rules (additive changes only), adds a contract with Lena ("send me a note before any ALTER on orders"), and adds a test that fails when a column's type changes in the capture stream. The integer-to-decimal change now goes through a planned migration: add a new column, backfill, switch readers, drop the old column.