Interview questions
Emitting signals
1Why is a counter safer than a gauge when metric scrapes are lost?
A counter only increases, so the next value carries the whole history, and the rate simply spans a longer interval. A gauge is a point sample: a missed scrape is lost, and anything faster than the scrape interval is never seen.
Read moreCounters survive loss, gauges do not
2The metrics bill doubled after someone added a user_id label. Why?
The number of time series is the product of every label's number of distinct values, so one unbounded label multiplies the whole store. Labels must be bounded; identities belong in logs or traces.
Read moreCardinality is the cost model
3Metrics, logs, traces and profiles: when do you use each?
Metrics for the shape of aggregate behaviour and for alerts, cheaply; logs for a specific event; traces for cause and effect across services; profiles for where resources go. They differ in how their cost grows and in what detail survives.
Read moreThe real taxonomy
4Should traces be sampled at the start of a request (head-based) or after it ends (tail-based)?
Head-based sampling is cheap and stateless, but drops most rare errors. Tail-based sampling keeps the interesting traces, but must buffer every span of every trace until it completes, memory = span rate × trace duration, and needs coordination between collectors.
Read moreSampling: before or after
5What is an exemplar?
A trace ID attached to a histogram bucket: a link from an aggregate metric to one concrete request you can inspect.
Read moreExemplars: the cheap join
6Why do log systems index labels but scan the message body?
A full-text index over free text costs without bound, since the number of distinct terms keeps growing, while real queries are a narrow label filter plus a search for a rare word over a short time window.
Read moreHow log stores are built
7What is the single most useful field in a query profile?
Estimated against actual rows for each operator. It measures the planner's error in estimating row counts, the usual root of bad plans.
Read moreQuery profiling
The state of the data
8Why does a job's progress bar stall at 95%?
A job finishes when its slowest task does, not on average, and the last tasks are the slow, skewed ones. Report the percentage of tasks done plus the longest-running task.
Read moreProgress and ETA
9How should you measure the freshness of a dataset?
As now minus the latest event time in the dataset, never as time since the last successful run. Using event time also reveals delays upstream.
Read moreFreshness
10A table is updated once a day. What does a 99.9% SLA mean for it?
Very little: there are only 365 chances a year, so one missed day is already 99.7%. Define data SLAs in their own terms, such as "ready by 09:00 on 99% of business days".
Read moreData SLAs have coarse arithmetic
11Consumer lag is small, yet the downstream dashboard is hours behind. Why?
Lag in offsets is not lag in time; lag summed across partitions hides one stuck partition; and the trend matters more than the level. Look at lag per partition, and in time.
12How do you detect data skew, and at what level should you warn?
From the distribution of partition sizes (cheap, after the fact) and a heavy-hitters sketch of key frequencies. Warn when the largest partition passes about 5 times the median.
Read moreSkew detection
From signal to action
13Why are most pages false alarms, even with a good detector?
Because real incidents are rare. With a 10⁻³ chance of an incident and a 1% false-positive rate, only about 9% of alerts are real. Alert on fewer things: symptoms that consumers can see.
Read moreAlerting is a base-rate problem
14What is a burn-rate alert?
It fires when the error budget is being spent at k times the sustainable rate over a window. A 14.4× burn over one hour spends 2% of a 30-day budget before it fires; 6× over six hours spends 5%.
Read moreSLOs, error budgets and burn rate
15Why can automatically restarting a service on out-of-memory be harmful?
It hides the leak, and can hold the system in a state that is broken but stable. Rate-limit and log it, escalate when it repeats, and track how often remediation runs as a metric.
Read moreRunbooks and auto-remediation
Active observation and capacity
16How can you tell that a file never arrived?
Nothing emits an event for an absence. Run a scheduled check against a declared expectation, add canaries to prove the path is alive, and keep the expectation up to date.
Read moreAbsence needs an expectationCanaries and synthetic transactions
Safe change, and watching the watcher
17How do you make a data deployment reversible?
Make visibility the last, atomic step: write to staging, audit it, then swap a pointer (write–audit–publish). Keeping the previous version as a blue-green copy gives instant rollback, at twice the storage.
Read moreWrite-audit-publishBlue-green tables
18Why must the alerting path be separate from the system it watches?
An observer that shares the system's fate fails with it and goes silent, and silence looks like health. Run alert evaluation, paging and a dead-man's switch outside it.
19Why should you capture the system's state before mitigating an incident?
Mitigation, such as a restart or a rollback, destroys the state that diagnosis needs. Capture first, then mitigate.
Read moreThe incident lifecycle
20Why is a blameless incident review an engineering choice, not just a cultural one?
People withhold detail when they fear sanctions, and their account is often the only record of state nobody instrumented. Blame degrades the sensor.
Read moreThe incident lifecycle