Summary and key takeaways
The reframe. Observability is a second machine whose output is a projection of the first. A system is observable only if you pay for that property. The design question is what to throw away, along four dimensions: time resolution, cardinality, population and retention. Three new constraints: information (you see a lossy projection), self-reference (the telemetry path is a pipeline with the same problems), and attention (humans do not scale with money).
Six families. Emission (four signals + change events), data-state observables, signal to action, active observation, economic and capacity observation, and safe change.
The signals, one sentence each:
- Metrics: cheap, independent of event rate, keep aggregate shape, lose identity. Counters survive loss, gauges alias, histograms fix resolution early, series count = product of label cardinalities.
- Logs: keep the specific record, cost grows with every event; index labels and scan bodies.
- Traces: keep causality, but only if every hop propagates context; sampling is forced (head = cheap and blind, tail = smart and costly); exemplars join metrics to traces.
- Profiles: sample the stack for where; off-CPU time matters for data systems; compare estimated and actual rows per operator.
- Change events: nearly free and resolve the most incidents.
Data observables. Services are correct per request; datasets are correct per arrival. Freshness is a property of the data (event-time based), never of the job. Absence needs an explicit expectation. Lag: watch the derivative and per-partition values. Skew: watch max/median (past about 5× you lose parallelism). ETA is wrong because the maximum task decides.
Alerting. It is a base-rate problem: at 10⁻³ incident rate and 1% false positives, precision is 9%. Page on symptoms, put causes on dashboards. Error budgets and burn-rate alerts (14.4× / 1 h = 2% of budget; 6× / 6 h = 5%) make detection speed a priced choice. Suppression (dedup, dependency grouping, ownership routing) fights fatigue. Auto-remediation must be rate-limited and counted.
Active observation. Canaries manufacture traffic to see absence, but prove only liveness along their own path. For cost, watch the derivative of unit costs, not totals; for capacity, watch a leading signal such as queue depth, not utilization.
The observer and its incidents. The telemetry pipeline must be lossy by design and its alert path must live outside the observed failure domain, with a dead-man's switch. Change events are the cheapest signal: "what changed in the last hour?" resolves more incidents than any detector. Capture state, then mitigate. Blameless reviews exist because blame degrades the sensor.
Equations and rules to remember:
| Rule | Meaning |
|---|---|
| Time to drain = lag ÷ (consume − produce) | undefined or infinite when consumption ≤ production; watch the slope |
| Precision = P(real) / (P(real) + P(false alarm)) | when real incidents are rare, even 1% false positives make 9% precision |
| Error budget = (1 − SLO) × window | 99.9% over 30 days = 43.2 minutes |
| Fault cost = damage rate × (detect + decide + repair) | shrink the cheapest term first |
| Headroom ≥ growth rate × lead time | 10%/month × 1.5 months ≈ 15% |
Patterns that repeat:
- The fault lives in what you discarded.
- Aggregation destroys the shape that contains the fault: the max, not the mean, hurts.
- Early binding fixes your questions: bucket boundaries, labels, structured fields, head sampling.
- Detection is cheap; suppression is hard.
- Every measurement that becomes a target stops measuring.
- The observer must fail independently of the observed, and never does entirely.