Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Snapshot expiration: history costs space

How much storage does retained history cost, and why does compaction raise it?

Retained history is every file that was ever superseded and is still inside the . If a fraction f of the table is rewritten per day and retention is R days:

RelationshipWhat retained snapshots cost →
share of the table rewritten per day
days of snapshots kept
light rewriting (5%/day)aggressive compaction (30%/day)
Data table
Storage ÷ live table size, by retention
Retention R (days)seriesStorage ÷ table size
0light rewriting (5%/day)1×
0.375light rewriting (5%/day)1.02×
0.75light rewriting (5%/day)1.04×
1.13light rewriting (5%/day)1.06×
1.5light rewriting (5%/day)1.07×
1.88light rewriting (5%/day)1.09×
2.25light rewriting (5%/day)1.11×
2.63light rewriting (5%/day)1.13×
3light rewriting (5%/day)1.15×
3.38light rewriting (5%/day)1.17×
3.75light rewriting (5%/day)1.19×
4.13light rewriting (5%/day)1.21×
4.5light rewriting (5%/day)1.23×
4.88light rewriting (5%/day)1.24×
5.25light rewriting (5%/day)1.26×
5.63light rewriting (5%/day)1.28×
6light rewriting (5%/day)1.3×
6.38light rewriting (5%/day)1.32×
6.75light rewriting (5%/day)1.34×
7.13light rewriting (5%/day)1.36×
7.5light rewriting (5%/day)1.38×
7.88light rewriting (5%/day)1.39×
8.25light rewriting (5%/day)1.41×
8.63light rewriting (5%/day)1.43×
9light rewriting (5%/day)1.45×
9.38light rewriting (5%/day)1.47×
9.75light rewriting (5%/day)1.49×
10.1light rewriting (5%/day)1.51×
10.5light rewriting (5%/day)1.52×
10.9light rewriting (5%/day)1.54×
11.3light rewriting (5%/day)1.56×
11.6light rewriting (5%/day)1.58×
12light rewriting (5%/day)1.6×
12.4light rewriting (5%/day)1.62×
12.8light rewriting (5%/day)1.64×
13.1light rewriting (5%/day)1.66×
13.5light rewriting (5%/day)1.68×
13.9light rewriting (5%/day)1.69×
14.3light rewriting (5%/day)1.71×
14.6light rewriting (5%/day)1.73×
15light rewriting (5%/day)1.75×
15.4light rewriting (5%/day)1.77×
15.8light rewriting (5%/day)1.79×
16.1light rewriting (5%/day)1.81×
16.5light rewriting (5%/day)1.83×
16.9light rewriting (5%/day)1.84×
17.3light rewriting (5%/day)1.86×
17.6light rewriting (5%/day)1.88×
18light rewriting (5%/day)1.9×
18.4light rewriting (5%/day)1.92×
18.8light rewriting (5%/day)1.94×
19.1light rewriting (5%/day)1.96×
19.5light rewriting (5%/day)1.98×
19.9light rewriting (5%/day)1.99×
20.3light rewriting (5%/day)2.01×
20.6light rewriting (5%/day)2.03×
21light rewriting (5%/day)2.05×
21.4light rewriting (5%/day)2.07×
21.8light rewriting (5%/day)2.09×
22.1light rewriting (5%/day)2.11×
22.5light rewriting (5%/day)2.13×
22.9light rewriting (5%/day)2.14×
23.3light rewriting (5%/day)2.16×
23.6light rewriting (5%/day)2.18×
24light rewriting (5%/day)2.2×
24.4light rewriting (5%/day)2.22×
24.8light rewriting (5%/day)2.24×
25.1light rewriting (5%/day)2.26×
25.5light rewriting (5%/day)2.28×
25.9light rewriting (5%/day)2.29×
26.3light rewriting (5%/day)2.31×
26.6light rewriting (5%/day)2.33×
27light rewriting (5%/day)2.35×
27.4light rewriting (5%/day)2.37×
27.8light rewriting (5%/day)2.39×
28.1light rewriting (5%/day)2.41×
28.5light rewriting (5%/day)2.42×
28.9light rewriting (5%/day)2.44×
29.3light rewriting (5%/day)2.46×
29.6light rewriting (5%/day)2.48×
30light rewriting (5%/day)2.5×
0aggressive compaction (30%/day)1×
0.375aggressive compaction (30%/day)1.11×
0.75aggressive compaction (30%/day)1.23×
1.13aggressive compaction (30%/day)1.34×
1.5aggressive compaction (30%/day)1.45×
1.88aggressive compaction (30%/day)1.56×
2.25aggressive compaction (30%/day)1.67×
2.63aggressive compaction (30%/day)1.79×
3aggressive compaction (30%/day)1.9×
3.38aggressive compaction (30%/day)2.01×
3.75aggressive compaction (30%/day)2.13×
4.13aggressive compaction (30%/day)2.24×
4.5aggressive compaction (30%/day)2.35×
4.88aggressive compaction (30%/day)2.46×
5.25aggressive compaction (30%/day)2.58×
5.63aggressive compaction (30%/day)2.69×
6aggressive compaction (30%/day)2.8×
6.38aggressive compaction (30%/day)2.91×
6.75aggressive compaction (30%/day)3.02×
7.13aggressive compaction (30%/day)3.14×
7.5aggressive compaction (30%/day)3.25×
7.88aggressive compaction (30%/day)3.36×
8.25aggressive compaction (30%/day)3.48×
8.63aggressive compaction (30%/day)3.59×
9aggressive compaction (30%/day)3.7×
9.38aggressive compaction (30%/day)3.81×
9.75aggressive compaction (30%/day)3.92×
10.1aggressive compaction (30%/day)4.04×
10.5aggressive compaction (30%/day)4.15×
10.9aggressive compaction (30%/day)4.26×
11.3aggressive compaction (30%/day)4.38×
11.6aggressive compaction (30%/day)4.49×
12aggressive compaction (30%/day)4.6×
12.4aggressive compaction (30%/day)4.71×
12.8aggressive compaction (30%/day)4.82×
13.1aggressive compaction (30%/day)4.94×
13.5aggressive compaction (30%/day)5.05×
13.9aggressive compaction (30%/day)5.16×
14.3aggressive compaction (30%/day)5.27×
14.6aggressive compaction (30%/day)5.39×
15aggressive compaction (30%/day)5.5×
15.4aggressive compaction (30%/day)5.61×
15.8aggressive compaction (30%/day)5.72×
16.1aggressive compaction (30%/day)5.84×
16.5aggressive compaction (30%/day)5.95×
16.9aggressive compaction (30%/day)6.06×
17.3aggressive compaction (30%/day)6.17×
17.6aggressive compaction (30%/day)6.29×
18aggressive compaction (30%/day)6.4×
18.4aggressive compaction (30%/day)6.51×
18.8aggressive compaction (30%/day)6.63×
19.1aggressive compaction (30%/day)6.74×
19.5aggressive compaction (30%/day)6.85×
19.9aggressive compaction (30%/day)6.96×
20.3aggressive compaction (30%/day)7.08×
20.6aggressive compaction (30%/day)7.19×
21aggressive compaction (30%/day)7.3×
21.4aggressive compaction (30%/day)7.41×
21.8aggressive compaction (30%/day)7.52×
22.1aggressive compaction (30%/day)7.64×
22.5aggressive compaction (30%/day)7.75×
22.9aggressive compaction (30%/day)7.86×
23.3aggressive compaction (30%/day)7.97×
23.6aggressive compaction (30%/day)8.09×
24aggressive compaction (30%/day)8.2×
24.4aggressive compaction (30%/day)8.31×
24.8aggressive compaction (30%/day)8.43×
25.1aggressive compaction (30%/day)8.54×
25.5aggressive compaction (30%/day)8.65×
25.9aggressive compaction (30%/day)8.76×
26.3aggressive compaction (30%/day)8.88×
26.6aggressive compaction (30%/day)8.99×
27aggressive compaction (30%/day)9.1×
27.4aggressive compaction (30%/day)9.21×
27.8aggressive compaction (30%/day)9.32×
28.1aggressive compaction (30%/day)9.44×
28.5aggressive compaction (30%/day)9.55×
28.9aggressive compaction (30%/day)9.66×
29.3aggressive compaction (30%/day)9.78×
29.6aggressive compaction (30%/day)9.89×
30aggressive compaction (30%/day)10×
Stated in 2 places →Principle: Retention is paid by the live path →

The non-obvious term is f: , so it makes f larger. Aggressive compaction and deep time travel multiply each other. You pay storage for every intermediate file size tier for the whole retention window. Tuning these two settings independently is a mistake.