Metrics: counters, gauges and histograms

Why do counters survive lost scrapes when gauges do not, and why can percentiles never be averaged?

A metric is a number sampled over time: orders per second, queue length, memory used. There are three basic kinds. The differences are not cosmetic. They decide what survives when data is lost.

Counters survive loss; gauges do not

A counter only goes up (total orders so far). A gauge is a value that moves up and down (current queue length).

A counter is monotonic (never decreases) by construction, and this single property is the whole design. Suppose the monitoring system reads the counter every 15 s and one read fails:

t=0s   orders_total = 1000
t=15s  (scrape lost)
t=30s  orders_total = 1090

Nothing is lost. The next value still carries the full history. The rate is (1090 − 1000) / 30 s = 3 orders per second. The calculation simply spans a wider interval.

A gauge is only a point sample of a continuous signal. A missed scrape is permanently lost. Worse, a phenomenon that is shorter than twice the scrape interval is aliased: it looks like noise, or like a slow trend that does not exist. (This is the sampling limit of Nyquist: to see a repeating pattern you must sample at least twice as fast as the pattern repeats.) Example: a queue that fills to 10,000 records for 5 seconds and empties again, once a minute, is invisible with a 60 s scrape. The sampled values look flat, or show a random small bump.

So the rule is: counters are loss-tolerant; gauges are limited by the sampling rate. Anything you care about should be emitted as an accumulator (a counter) and differentiated at read time (turn into a rate when you query), not accumulated at write time.

# Emit an accumulator, compute the rate when you read it.
orders_total.inc()                      # in the application: one line per order

# In the query language of a monitoring system (PromQL-style):
#   rate(orders_total[5m])             # orders per second over the last 5 minutes

Histograms fix their resolution before the question

You cannot store every observation (a service handling 20,000 requests per second would produce 1.7 billion latency values per day). And the mean destroys exactly the information you need: a mean of 120 ms can hide a tail of 5 s. This is the variance beats volume law: the spread is the problem, and the average hides the spread.

So a histogram counts observations in buckets:

But the bucket boundaries are fixed when you emit (this is early binding: you decide the structure before you know the question). Ask for p99.9 on a system whose largest bucket is 1 s, and the answer is "somewhere above 1 second", permanently. The information was never stored.

There is an operational corollary: pre-bucketed histograms can be summed across instances, pre-computed percentiles cannot.