Observation is a second machine

Which three constraints shape every observability system, and why does exactness stop mattering?

Every system up to this point in the stack transforms data. Observability builds a second, smaller machine whose output is a description of the first. That inversion is the key difference. The product of this machine is not data. It is a reconstruction of internal state from an outside view.

Control theory gives the precise word. A system is observable if you can infer its internal state from its outputs alone. Nothing guarantees this property. A pipeline does not become observable by itself. You have to pay for it deliberately.

Three constraints appear here that did not exist when we only built the pipeline itself.

Constraint 1: the information constraint

You never see the system. You see a lossy, delayed, aggregated projection of it. (A projection is a view from one side, like a shadow of an object. Many different objects can cast the same shadow.)

Why lossy? Because full fidelity is impossible to afford. Recording every state change of every operator is a second copy of the workload. So the real design question is what to throw away. Every mechanism here is a choice about which dimension of fidelity to sacrifice:

You are building a compression of system state. The loss function of that compression is: "can I still reconstruct the fault?"

Constraint 2: the self-reference constraint

The telemetry path is itself a data pipeline: it ingests, partitions, indexes, compacts, retains and queries. So it inherits every problem of data pipelines: it has a cardinality explosion problem (Emission), a small-files problem, a hot-partition problem, and a cost problem.

Worse, it often shares machines and networks with the system it watches. That breaks the assumption that makes it useful. An observer that fails together with the observed produces silence exactly when you need a signal, and silence looks the same as health.

Constraint 3: the attention constraint

This is the first layer where the final consumer is a human, and the limiting resource is human attention. Attention has a fixed rate, it does not stretch, and it gets worse under load. Machine capacity scales with money. On-call capacity does not.

Almost every operational problem here is the same collision: a fixed rate (human attention) meets a growing system. An alert system that fires 10 pages a week works. The same system at 10 times the size fires 100 pages a week, and the people are the same size. This is why the interesting mechanisms are suppression mechanisms (deciding what not to say) and not detection mechanisms. Detection has been cheap for twenty years. Deciding what not to say is the hard part.

What stops mattering: exactness

This is the one data pipeline in the whole stack where approximation is the default and nobody objects:

• sampled traces (you keep a few percent);
• histogram buckets instead of raw latencies;
• probabilistic estimates of distinct counts, and sketches;
• 10-second resolution of a phenomenon that lasts a microsecond.

Durability also relaxes. Losing an hour of metrics is an inconvenience, not data corruption. This is why sketch structures (which in a database are an optimization) are the normal representation here.

The governing question

Read each part of the question as a constraint: smallest (cost), reconstructed (information), fast enough (the fault keeps doing damage while you wait), while degraded (the observer must survive the incident).