Summary and key takeaways
The reframe. Maintenance is the first work in the stack that nobody requested, that adds no information, and that changes no answer. Correctness vanishes as a constraint. What replaces it is scheduling and economics with no deadline, under entropy from continuous write.
Three degradations, none a bug:
- File sizes fall below optimum (files = writers × partitions × commits, not data volume).
- Sort order decays (arrival order ≠ query order).
- Metadata grows without bound (immutability means append only).
The governing question. Who pays for the gap between arrival order and read order: the writer, every reader, or a background job, and what is the break-even?
The four families, one sentence each:
- Layout restoration: rewrite cost paid once against read savings paid per query; best file size is a multiple of F × b (128 MB-1 GB).
- Reclamation: reachability is computed, liveness is guessed; the retention window is a bound on query duration.
- Derived-state refresh: statistics and indexes are claims about a moment; mergeable ones are cheap; staleness fails silently.
- Semantic migration: same machinery, but it changes the data model or the answers; legal deadlines are external.
Key equations:
| Equation | Meaning | Seen in |
|---|---|---|
| overhead = F / (F + s/b) | share of scan time lost to per-file cost | best file size, small files |
| rewrites = log_g(s_target / s_initial) | compaction write amplification | bin-packing |
| files touched ≈ N^(1 − k/d) | pruning with a space-filling curve | Z-order clustering |
| storage ≈ size × (1 + f·R) | storage with history | snapshot expiration |
| T = √(2C / r) | best maintenance interval | compaction, checkpoints, stats refresh |
| rewrite when d × future reads > 1 | delete compaction | tombstone merge |
Patterns that repeat:
- Granularity mismatch ⇒ amplification (small files, delete by file, tiering requests).
- Immutable layer ⇒ indirection + collector (SSD GC, LSM compaction, table maintenance).
- Costs are moved, not removed (merge-on-read vs. copy-on-write).
- Small files are fatal mainly through planning, not GET price.
- Unrequested work has no advocate, so make it automatic.
- Stale derived state fails silently and bills someone else.
- Narrow conflict predicates make background work possible.