What observability forbids
What can never be observed?
A closing request from Omar
Omar asks Nadia for one page: "What can we never do, what do we do instead, and what does each choice cost? I need to justify the budget." This article is that page.
What the layer forbids
- You cannot observe absence without a prior expectation. Every missing-data detector is a scheduled comparison against a declaration that can itself go stale.
- You cannot reconstruct causality from independent observations. It must be propagated in-band, and one gap severs it.
- You cannot recompose percentiles. Any architecture that pre-aggregates them has destroyed the number.
- You cannot carry unbounded cardinality in an aggregate signal without paying the full product.
- You cannot detect faster than the emission interval plus the collection delay. This sets a floor on MTTD that is independent of how good your detectors are. (If metrics are scraped every 60 s and take 30 s to arrive, you cannot know in less than ~90 s.)
- You cannot make a shared-fate observer reliable.
- You cannot have full fidelity and negligible overhead. Full fidelity is a second copy of the workload.
The escapes, and what each costs
| Forbidden thing | Escape | Price |
|---|---|---|
| Full-fidelity emission | Sampling (head/tail), aggregation, downsampling tiers | Rare events disappear; tail sampling needs buffering and cross-collector coordination |
| Unbounded cardinality in metrics | Move identity to logs/traces; exemplars as the join key | Higher per-event cost; only sampled identity survives |
| Recomposing percentiles | Emit bucketed histograms, merge buckets | Resolution fixed at write; unanswerable questions later |
| Causality without propagation | Trace context in-band | Every hop must be instrumented; coverage is all-or-nothing |
| Observing absence | Canaries, heartbeats, dead-man's switches | Contamination downstream; coverage limited to the canary's path |
| Detection speed vs. false alarms | Multi-window burn-rate alerting | Explicit, quantified budget consumed before firing |
| Alert volume vs. fixed attention | Symptom-based alerting, dependency grouping, ownership routing | Depends on lineage and ownership metadata being correct |
| Irreversible data deploys | WAP, blue-green, shadow diff | 2× storage or 2× compute; diffs must handle non-determinism |
| Shared-fate observer | Externalize the alerting path only | Two systems, two operational burdens, split context during incidents |
| Attribution on shared resources | Tag-and-join, or isolation per tenant (attribution) | The allocation rule is a choice that creates incentives; isolation gives up pooling efficiency |
Two laws argued only here
Law B. Cost of a fault = damage rate × (time to detect + time to decide + time to repair). Three separable terms with very different marginal costs. Detection is usually the cheapest to shrink and the most neglected. Decision time is what runbooks attack. Repair time is what blue-green and rollback attack. Optimizing repair while detection takes six hours is spending in the wrong term.
Law C. Every measurement that becomes a target stops measuring. This is Goodhart's law and the first truly organizational law here. Uptime targets produce redefinitions of "down". Alert-count targets produce suppression rather than fixes. Chargeback produces cost pushed onto unattributed shared infrastructure. Freshness SLAs produce jobs that publish empty partitions on time. The mechanism is always the same: the metric is a proxy, and optimizing a proxy is cheaper than optimizing the real thing.
Law B is the one to carry forward hardest. It is the only frame that lets you argue about observability spending quantitatively: it asks what a measurement is actually worth.