The telemetry pipeline is a data system

Why must telemetry be lossy by design, and why must alerting not share a failure domain?

The path from the application to the dashboard is agent → collector → store. An agent runs next to the application and sends data out. A collector receives, batches and forwards data. The store keeps it and answers queries. This path inherits the whole data-engineering stack: ingest, partition, index, compact, tier (hot, warm, cold storage), expire. Two properties are non-negotiable.

It must be lossy by design

When the collector is saturated, the correct behaviour is to drop telemetry, never to apply backpressure into the application (slow the application down until the collector catches up). An observability system that stalls the workload has turned monitoring into an outage.

A failure domain is the group of things that fail together (one rack, one cluster, one cloud region). Putting the alerts in the same failure domain as the app is a contradiction of the usual shared-fate reality: the cluster dies, and the monitoring dies with it. This is what happened to Nadia when the cluster failed.

The practical resolution everyone converges on:

Externalize the small critical path: alert evaluation, paging, and a dead-man's switch (an alert that fires when telemetry stops arriving, so silence itself is detected). For example, the application sends a heartbeat every minute to an external service, which pages if no heartbeat arrives for 5 minutes.
Let the large, expensive analysis path share infrastructure.

Cost typically lands in the range of 5–15% of infrastructure spending. This is engineering reality, not a law, and it varies by an order of magnitude depending on log discipline.

Case, resolvedConstructed case · Nadia at Brightcart

Forty dashboards green, revenue 12% low

For three days finance used revenue numbers about 12% too low while every job ran successfully, and a weekend network fault took the monitoring down with the cluster it watched. Where it started: A second machine →

The projection Brightcart had chosen was "did the job run, and did it use normal CPU?" That projection keeps nothing about the data. A source feed had been sending half-empty files for three days. The job ran successfully over fewer rows. In the "shadow" cast by the monitoring, a job that processes 50% of the rows looks exactly like a healthy one.

The weekend outage had the opposite cause. The observer shared a failure domain with the observed, so the cluster's failure silenced the alerts about it. Nadia moves alert evaluation and paging to an external service and adds a heartbeat dead-man's switch. When the cluster fails again in a drill, the heartbeat stops and a page arrives in 5 minutes.

Nadia writes one sentence on the whiteboard: "The fault lives in what we discarded."

The telemetry pipeline is a data system

It must be lossy by design

The alerting path must not share a failure domain with the observed system