Scenarios

Job success and CPU say nothing about the data itself.

Freshness on event time, alerting when it exceeds a threshold derived from the SLO; a volume band compared with the same weekday; lag per partition, with its trend; and skew, with the largest partition under 5 times the median.

A declared expectation for each feed, with an owner, and canaries at source and sink to prove the path is alive.

Page only when the table is stale or too small at 08:00, or ingestion has stopped, with a burn-rate rule for the SLO "ready by 09:00 on 99% of business days". Causes go on dashboards.

Write–audit–publish, auditing against the payment provider's total.

More checks mean more maintenance, and every expectation can go stale. Start with freshness, volume and write–audit–publish.

Suspect cardinality. Count series per metric and compare with last week, looking for a new label with unbounded values: a user ID, a request ID, or a URL path containing an ID. 5,000 series × 200,000 values = 10⁹ series.

Drop the label at the collector.

A series limit per metric, enforced in the pipeline (drop and count violations, never apply backpressure to the application); a code-review rule that labels be bounded and enumerable; and a change-event marker for each release, so correlation is quick. Move identities into structured logs or traces, linked by exemplars.

Per-user aggregates leave the metrics; per-user questions become log queries, which cost more and are sampled.

It is a problem of base rates and attention: alert volume grows with the system, and attention does not. A better detector cuts false positives by some factor, but when real incidents are 10⁻³ of checks, even a 0.1% false-positive rate gives only about 50% precision.

1. Page only on symptoms consumers can see.
2. Multi-window burn-rate alerts, which say how much budget is spent before they fire.
3. Group alerts by dependency using lineage, so one root cause pages once (this depends on correct lineage).
4. Route alerts by ownership.
5. Test severity against action: "look at it tomorrow" is a ticket, not a page.

Anomaly models can learn a broken state as normal, and move the work into maintaining models. Measure pages per week and the share that needed action, but do not make "fewer pages" a target on its own, or people will suppress real alerts.

Offset lag is a level, and a sum. The cause may be one stuck partition (the maximum is what hurts), or producers outpacing consumers (the trend). Time to drain = lag ÷ (consume rate − produce rate), which is infinite once consumers are no faster than producers.

Lag in time (the age of the oldest unconsumed record) per partition; the trend of lag; the largest partition against the median; and the freshness of the output table, which is the actual symptom.

Page on freshness, or on time lag rising fast enough to breach the SLO within some horizon ("will breach in 40 minutes at the current slope").

Series per partition cost partitions × consumer groups; keep that number bounded and enumerable.

Tail sampling must buffer span rate × trace duration = 50,000 × 10 = 500,000 spans in flight. At about 1 KB per span, that is about 500 MB, so one collector is not enough, and every span of a trace must reach the same collector (route by trace ID). At 200 MB each you need at least 3 collectors, more with headroom.

Head-sample the bulk at, say, 10% (cheap and stateless), and add tail rules only for high-value traffic such as errors and slow requests; or rely on exemplars, so every histogram bucket still carries an example.

Head sampling drops rare errors; tail sampling costs memory and a coordination layer, which belongs to the telemetry system and must drop when full rather than stall the application.

Tracing every record does not fit; use sampled records and lineage at the level of datasets.

1. A shadow run for 2–3 nights on identical input, comparing aggregates and distributions within a tolerance, not individual rows, because of non-determinism. Costs double compute for that window.
2. Write–audit–publish for every publication: write to staging, audit (a band for row count, null rates, totals against an independent source), and publish with an atomic pointer swap, which needs atomic metadata commits from the table format.
3. Keep the previous version as a blue-green copy for N days, for instant rollback, at double the storage for that table.
4. Emit a change event at deploy time on every freshness and volume chart.

Shorten the time to detect it: a fault's cost ≈ rate of damage × (time to detect + decide + repair).

Audits too weak, proving only that the output looks plausible; a shadow period that misses month-end; and consumers reading the staging table by mistake, so restrict its permissions.

Telemetry is ingested, partitioned, indexed, compacted, tiered and expired like any other data.

It must be lossy by design: when saturated, drop telemetry and count the drops; never apply backpressure to the application. And the alerting path must not share a failure domain with what it watches.

A small external critical path (alert evaluation, paging, and a dead-man's switch that pages when the heartbeat stops for, say, 5 minutes) and a large shared path for analysis. Budget about 5–15% of infrastructure cost, depending on logging discipline.

Two systems mean two operational burdens and context split during incidents; the external path needs its own heartbeat check; and where you draw the line between external and shared decides which incidents remain invisible.