Which clock keys the partition
Should files be partitioned by event date or by ingestion date, and what does each choice cost?
This is where the topic meets physical storage, and it is the decision with the widest effects: it touches every later query, rewrite and report.
Partition by ingestion date
- Writes stay append-only: new records always go to today's folder. The past is immutable.
- Backfills (re-loading old data) are easy to reason about. Snapshots (saved versions of the table) stay small.
- But every query by event time must scan a range of folders wide enough to cover the skew tail, forever, on every query. A query for "5 March" must also read the folders of 6 March, 7 March, and so on, because late records for 5 March are in those. You pay the tail at read time.
Partition by event date
- Queries prune correctly: "5 March" reads only the 5 March folder.
- But late data changes old partitions. That means rewrites, compaction churn (background rewriting of small files into big ones), growth of snapshots, and consumers who already read a partition they believed final. The past becomes mutable, which is exactly the property that immutable storage was designed to remove.
The usual resolution: do both
- Keep an ingestion-partitioned raw zone for auditability and replay.
- Build an event-partitioned curated zone, which is rewritten during a bounded lateness horizon and then frozen.
The horizon at which a partition is declared final is a published contract. Almost nobody publishes it, which is why downstream restatements surprise people.