Partition evolution

Why are partition changes different from compaction, and what does evolving a spec in metadata cost?

CaseConstructed case · Nadia at Stackwise

Delete one customer from an immutable table

Three requests arrive in one week:

A customer in Europe writes: "Please delete all my data within 30 days." Stackwise's table keeps 30 days of time travel, and files hold millions of customers each.
Finance asks to move data older than a year to cheaper cold storage. Nadia tries it on a test set, and the bill goes up.
The product team wants to change partitioning from by day to by day and region, for 3 years of data.

All three are called "maintenance." But these are not like compaction. These change the data model, or even the answers.

These jobs use the same machinery as the earlier families, which is rewriting bytes in the background. But they are not semantics-preserving, and one of them changes query answers.

How it ends →

With directory-based layouts (for example /date=2025-01-05/part-000.parquet) the partition scheme is written into every file's path, so changing it means rewriting everything. Table formats that record partition values per file let old and new specs coexist; deciding a spec change and planning across specs is schema evolution's subject. What remains here is the bill: evolution is free going forward, but repartitioning history is a full rewrite, bounded by the number of bytes. There is no clever path. (Nadia's third request would cost her 3 years of rewriting; she chooses to apply the new spec to new data only.)