Recording is irreversible; distributions, not averages

Why must you keep the shape of the data before you know the question?

CaseConstructed case · Ingrid at Bellwether Tickets

Mean latency 12 ms, and fans timing out

Ingrid now has correct load. She hands the raw numbers to Yuki, who owns the dashboards. Yuki's dashboard shows mean latency 12 ms for the seat-hold service. A second panel shows the average of the per-shard p99 values: 60 ms. Both look fine. Yet fans report time-outs, and the support queue is growing.

The Tessera slide from the vendor says "peak throughput 180,000 holds/s". During a recent retry storm, the dashboard showed the service at full speed and healthy, while almost no fan managed to get a seat. Last, a CPU profile of one slow request looked completely flat, and nobody knew what to optimize.

Every number on these screens is correct as arithmetic. Each one answers a question nobody asked. This article explains why.

How it ends →

Why recording is irreversible

The load is now correct. This article is about what you do with the observations. The new difficulty is that the failures here are irreversible. A bad experimental design can be re-run. A bad load model can be fixed and re-run. But an observation stored as a mean is a distribution that no longer exists, and no later analysis can bring it back.

The governing constraint is about information. Every aggregation (every step that replaces many numbers with fewer numbers) is a lossy compression. You choose it before you know which question you will ask later. The compression that fits in your metrics budget is almost never the one that answers the question.

The second constraint is that measuring changes the thing measured. In earlier topics, observation was free because nothing observed itself. Here, the observer runs on the observed machine and uses its CPU, cache and memory.

Aggregation is a decision about which future questions you are permanently unable to answer.

This is the principle "aggregation is lossy and irreversible": summarize only after you know the question, and keep enough raw or mergeable data to ask a new one.

Distributions, not averages

The mean (the sum divided by the count) is the wrong summary because latency distributions do not have the shape that a mean describes. Service times in real systems are heavy-tailed (a small fraction of requests are far slower than the rest) and usually multi-modal (they have several separate clusters). Typical causes of clusters are: cache hit versus cache miss, fast path versus spill to disk, local replica versus remote replica. Each cluster is a different mechanism, because systems are piecewise (they behave differently in different regions). So a mean is an average across mechanisms that never occur together. It describes a case that never happens.

The practical consequence has a direction. With a heavy tail, the mean sits well below the tail but well above the typical (most common) value. It therefore overstates the typical experience and understates the bad one. Two systems with identical means can differ by 10x at p99. Optimizing the mean optimizes the wrong mechanism.

What to report instead: enough of the shape to see the clusters. Use the median (p50), p90, p99, p99.9 and the maximum. When you compare two systems, show the full CDF (cumulative distribution function: for each latency t, the fraction of requests that finished within t). Plot it on a log-odds or quantile axis, where the tail differences are visible. A linear-axis histogram flattens exactly the region you care about.

One correction to a common belief: higher percentiles are not simply "more informative". The estimate of each percentile has a sampling error that grows toward the tail. A p99.9 computed from 10,000 samples rests on roughly ten observations (the slowest 0.1% of 10,000 is 10 values), so it is dominated by noise. The percentile you can report is limited by your sample count. Claiming p99.99 from a five-minute run is an arithmetic failure, not rigour.

The extreme case: a number too rare to observe

Durability figures push that rule to its limit. "Eleven nines" means an expected loss of 10⁻¹¹ of your objects per year, and no observation can check it. If you watch n object-years and lose nothing, the most you can claim is that the yearly loss rate is probably below about 3 ÷ n (at 95% confidence, a zero count bounds the rate near 3 ÷ n).

So a durability number is a model output, not a measurement, and measuring it goes wrong in a specific way: an empty loss log is read as confirmation, when it is only a weak upper bound. What can be measured are the model's inputs, and they are where the number is decided: the device failure rate in your own fleet, the rebuild time under production load (not on an idle cluster), how often failures arrive together (a rack, a power domain, a bad firmware batch), and whether a restore actually works when you run one. Each of those is an ordinary measurement with a configuration, a noise floor and a scope. The nines are only as true as the least-measured of them, usually the correlation.