Partition spec evolution
What must be true for a table's partitioning to be changeable?
Partitioning divides a table into pieces by the value of a column (for example, by day), so a query for one day can skip the other days. This is a physical commitment: it decides where data lives. So changing it retroactively means moving files.
The mechanism that makes evolution possible at all is to record the partition spec per file (or per snapshot), not as a single fact about the whole table. The planner then applies whichever spec was in force when each file was written.
Three preconditions
A system that lacks any one of these cannot offer partition evolution, however much engineering is spent.
- The partition value must be a declared function of a column, such as
day(ts)orbucket(id, N), rather than an independent column that the writer sets. Otherwise the relationship between the data and its location cannot be verified, and "re-specifying" it means nothing. - The spec must be recorded in metadata, not encoded in a path. Path-encoded partitioning (folders like
/date=2025-03-04/) is position identity again: the meaning is hidden in a place the reader has to guess, with all the failure modes of identity by position. - Old and new specs must coexist at plan time. The planner must handle both in one query.
The consequence
A query across the change boundary prunes (skips irrelevant data) well on new files and less well on old ones, with two pruning paths in one plan. Correctness is preserved. Performance becomes era-dependent.
Partitioning is evolvable only if it is a declared function of a column recorded per file. Otherwise location is identity, and identity cannot change in place.