Downsampling, rollups and retention tiers

Which aggregates can be rolled up, and why degrade resolution rather than delete?

Downsampling, rollups, and what may be precomputed

A year of one-second data is 365 × 24 × 3,600 = 31,536,000 points per series, about 31 million. A dashboard shows about a thousand pixels. Reading 31 million points to draw a thousand is four orders of magnitude of waste (31,000,000 / 1,000 = 31,000 times), repeated on every refresh.

So you precompute. Store one-minute, one-hour and one-day aggregates next to the raw data (a rollup), and route each query to the coarsest resolution that satisfies it (a planner decision of the same kind as rewriting a query to use a materialized view). This is precomputation versus compute-on-read, with the familiar break-even: pay once at write time, read many times.

The mechanism that actually matters is which aggregates survive this, and it is not a storage question at all.

Percentiles do not compose, and people routinely violate that rule. This is the algebra rule in its cleanest form: what can be rolled up is determined by the algebra of the aggregate, not by the storage design. The same property decides partial aggregation in distributed queries, combiners in BSP, and mergeable index structures in vector search. These are four places where one algebraic question decides the architecture.

The escape is to store a sketch (a small approximate summary of a data set) instead of a scalar. A HyperLogLog merges, so unique counts roll up. A t-digest or KLL sketch merges, so quantiles roll up. You replace a non-mergeable exact value with a mergeable approximate one. You trade exactness for composability. That trade is worth naming explicitly, because it is not obvious that those are the two things being exchanged.

Note the asymmetry with the rest of this mechanism: this is a two-sided approximation (errors in both directions) with no refine stage available, because the raw data it would refine against may already have been deleted. The one-sided rule says this is acceptable only where no exact answer is owed, and monitoring is exactly such a place.

One more mechanism is easy to miss and causes many wrong dashboards. Downsampling is a choice of which function reduces the points: min, mean, max and last all produce different, equally valid-looking pictures. A mean hides the spike you were monitoring for. Storing several reductions per bucket is the standard answer, and it is a small instance of the same materialization trade (storing precomputed results at the cost of space).

One line: rollups are only valid for aggregates whose algebra composes. The escape from non-composability is a mergeable sketch, which buys composability with exactness.

---

Retention tiers by resolution

Retention is normally a binary policy: keep data, or delete it after N days. Time series allows something better, because a coarse version of old data is genuinely useful while the fine version is not.

The value of resolution decays with age much faster than the value of the data itself. Yesterday's incident needs second-level detail. Last year's capacity trend needs daily points. So keep raw data for days, minute data for weeks, hourly data for months, daily data for years. The storage profile becomes nearly flat with age instead of growing linearly, because each tier is five to fifteen times smaller than the one before.

Keeping raw data for five years would need 5 × 31,536,000 = 157,680,000 points, about 240 times more.

This is a genuinely distinct idea, because retention elsewhere has no such lever:

Tiered storage (moving old data to slower, cheaper media) degrades access latency with age. This degrades information content with age. Both are the same shape of answer, matching the cost of keeping something to the declining value of having it, but they are different axes. Time series is one of the few places where the second is available. Other examples: image and video transcoding tiers; sampled traces kept at full fidelity for a day and at 1% forever; log aggregation into counters. The common structure is that the value of fidelity decays faster than the value of existence. The common price is that the degradation is irreversible and made before you know the question.

The costs are real and usually understated. The decision is irreversible: once the raw points are gone, no query can recover them, and the question you will want to ask in six months is not the one you designed the rollup for. Non-composable aggregates make it worse. If you did not store the right sketch at the right resolution, that question is permanently unanswerable. So the retention tier is effectively a schema decision, made at write time, about which questions remain askable. It is early binding with an unusually long and unforgiving horizon.

One line: the value of resolution decays faster than the value of age, so degrade rather than delete, and accept that you are deciding now which questions stay answerable later.