The application granularity ladder
Why does the cost of applying a change follow the substrate's granularity rather than the change's?
Detection granularity and application granularity are independent choices. Mismatching them is the most common design error in this area. Consider three ways to apply a row-level change.
| Detect at | Apply at | Name | Cost | Trade-off |
|---|---|---|---|---|
| row | partition | rewrite touched partitions whole | scales with partition size | cheap, safe, wasteful |
| row | file | copy-on-write: rewrite every file that contains a changed row | scales with files touched, not rows changed | one row spread over 1,000 files costs 1,000 file rewrites |
| row | row | merge-on-read: append a delete marker (a delete vector) plus the new row | pays at read time, and again at compaction (the background merge that cleans up) | fast writes, slower reads, cleanup debt |
Every merge into a target also has a fixed cost per commit, and that derives a natural micro-batch cadence and therefore a freshness floor. (Example: if each commit has a fixed cost of 30 s of metadata work and you want the overhead below 10% of the time, you should commit at most every 300 s, so your data cannot be fresher than about 5 minutes.)
So the substrate's mutation strategy decides the merge penalty c′ in the break-even equation at the start of this mechanism. This closes a loop: the economics of incremental processing are decided at the storage layer, long before the pipeline design.
In one line: the cost of applying a change scales with the granularity of the substrate, not the granularity of the change. This is granularity mismatch, paid in dollars.