Compression as the write path

How do time series compress to about a byte per point, and why does that make chunks immutable?

General-purpose compressors (such as gzip) work on time series, but badly compared with what is available, because they do not know the data is a sequence of near-constant numbers. Two encodings exploit the two facts true of essentially all time series (the best-known design is called Gorilla, from a published in-memory time-series database):

• Timestamps are near-periodic, so delta-of-delta (the difference of the differences between timestamps) is usually zero and costs one bit.
• Values change slowly, so the XOR of two consecutive floats cancels the exponent and the high bits, leaving a short run of differing bits; a constant value costs one bit.

How each encoding works, bit by bit, is encoding's subject (the float half). What the workload adds is how well they pay here, and what they do to the write path.

Together these bring a 16-byte point to roughly 1 to 2 bytes on well-behaved data. This is the number that makes the whole family viable. For Haatly: 1 TB of raw points at 16 bytes per point is about 62.5 billion points. At 1.5 bytes per point that becomes about 94 GB, a reduction of 16 / 1.5 ≈ 10.7 times.

Three things about this matter more than the encodings themselves.

It is stream-shaped. Each point is encoded against its predecessor. Compression happens at ingest, one point at a time, with no buffering. Columnar block compression, in contrast, needs a whole block before it can start. Compression moves from a background batch operation to an inline write-path operation, which is what makes it compatible with a memory-resident window of recent data.

It makes the data effectively immutable in chunks. A sequence of variable-length, chained-delta encodings cannot be randomly accessed or updated in place. You must decode a chunk from its start. So the chunk becomes the unit. Chunks are closed and frozen, and late-arriving or corrected data requires rewriting a whole chunk. This is the familiar shape of immutable storage (an indirection layer plus a garbage collector) arriving by a different route than usual: here immutability is a consequence of the encoding, not a storage policy.

The compression ratio is a property of the signal, not of the system. Regular intervals and slow drift give about 10 times. Jittery timestamps and noisy or high-entropy values give 2 times or worse. So capacity planning is workload-dependent in a way that is easy to get badly wrong, and the failure is quiet: the system does not break, it just stops fitting in memory. Variance costs more than volume again: it is the second place in this article where the harmful input is irregularity rather than volume.

One line: difference until the residual is zero and XOR until the bits cancel. Compression becomes a write-path operation, and the chunk becomes immutable as a side effect.