Incremental checkpoints and recovery time
What does incremental checkpointing require, and how long does catch-up take after an outage?
Incremental checkpointing
A full snapshot of 500 GB every 60 seconds needs 500 GB ÷ 60 s = 8.3 GB/s of sustained write bandwidth just for state. Nobody has that. So you upload only what changed since the previous checkpoint.
This is possible only if the backend's storage units are immutable. If files never change, a file that existed in the previous checkpoint is still valid and need not be uploaded again. That is why LSM-structured state backends and incremental checkpointing arrived together: the LSM's sorted files are immutable.
The consequence: a checkpoint becomes a set of references to files shared across many checkpoints. So you now need reference counting and file garbage collection over the checkpoint history: an immutable substrate grew an indirection layer and a collector.
It also means:
- Recovery cost is no longer bounded by the size of one checkpoint. A restore may need files from many older checkpoints.
- A long chain of incremental deltas needs periodic consolidation, which is compaction under another name. This is the granularity-mismatch law: the size of a change (small) does not match the size of a file (large), so amplification results.
Interval arithmetic: recovery time and headroom
Recovery time is roughly
Restore is easy to estimate. Catch-up is the part people get wrong: the lag built during the outage drains only at replay rate minus arrival rate, so as the job's load nears its capacity the catch-up time goes to infinity (the catch-up condition). Ana's job runs at 90% of a 100,000 records/s capacity: a 10-minute outage leaves 54 million records that drain at 10,000/s, so 90 minutes of catch-up. Headroom is not slack. It is the recovery budget.