Format migration

Why is every format migration dual-write, cut over, drop?

Readers and writers upgrade at different times, so migration is always dual-write, cut over, drop:

Write both the old and the new format.
Move readers to the new one.
Stop writing the old one and delete it.

This costs 2x during the window, and the risk is concentrated at the flip. It has the same structure as building an index online and as rolling out a schema change.

Case, resolvedConstructed case · Nadia at Stackwise

Delete one customer from an immutable table

In one week a customer asked for erasure within 30 days, a cold-storage test raised the bill, and product asked to repartition three years of data by day and region. Where it started: Semantic migration →

The legal request. Her design has no per-customer keys, so she must use the rewrite path. She finds the files that hold the customer's rows with the manifest min/max statistics, writes new files without those rows, and expires every snapshot that referenced the old files within the 30-day window. This also forces her to cut time-travel depth to 14 days, so that no old snapshot can bring the data back. She also proposes that new tables use per-customer encryption keys, so the next request costs a key deletion.
The cold-tier bill. Her test set was full of small files (each tiering request is per object) and most of it was read again within 2 months, less than the 3.5-month break-even. She tiers only large compacted files, and only partitions older than a year that have had no reads for 6 months.
Repartitioning. She applies the new partition spec to new data only; the planner handles both specs.