Observability & Operations
What is the smallest description of a running system from which we can find a fault, fast enough to matter, at a cost we can still pay while the system is failing?
Who this mechanism is for: a developer who builds data pipelines or backend services, has seen a dashboard or a log file, and has been woken by an alert, but has never studied why monitoring systems are built the way they are. You do not need to know Prometheus, OpenTelemetry or any other tool. Every technical term is explained the first time it appears, and again in the glossary at the end.
The story we will follow
Nadia is a data engineer at Brightcart, an online grocery company. Every order creates an event. The events travel through a message queue (Kafka) to an hourly Spark job. The job writes a clean orders_clean table and then a daily_revenue table. The finance team reads daily_revenue on a dashboard, and it must be correct by 09:00 every morning. Nadia's manager, Omar, also leads the on-call rota (the list of people who must respond to alerts at night).
Nadia will meet five problems, one at the start of each of the first five parts below. Each problem looks like a bug in the pipeline. Each one is really a problem of observation: the pipeline is fine, or broken, and nobody can tell. At the end of each article, we go back and solve it with the part's mechanisms. A short last part closes this mechanism with what observation can never do.
This is the main idea of this mechanism:
How this mechanism is organized
- 1
A second machine
Why is observing a system a different kind of engineering from building it, and what are the three constraints?
CaseForty dashboards green, revenue 12% low
- 2
Emission
What do metrics, logs, traces and profiles each keep and throw away, and what does each cost?
CaseOne label doubled the monitoring bill
- 3
Data-state observables
How do we observe *data* (freshness, lag, skew, progress) and not only services?
CaseA green job over empty input
- 4
Signal to action
Why are alerts mostly noise, and which mechanisms (budgets, burn rates, suppression, runbooks) fix it?
Case140 pages in a week, and the real one slept through
- 5
Active observation
How do we observe what does not report itself?
CaseA file that never arrived, and nothing moved
- 6
What observability forbids
What can never be observed, what are the escapes, and which laws repeat?
CaseWhat Brightcart will keep, and what it costs
After that: a summary, hands-on labs, interview questions and a glossary.
One request: watch the arithmetic. This mechanism has small calculations (a base rate, an error budget, a fan-out). They show that most observability problems are not about tools but about numbers you can compute before you buy anything.
What do I do, and what does each option cost?
Can I see it myself?
Counter vs. gauge under lost scrapes
A counter and a spiky gauge scraped every 15 s and 60 s with 30% of scrapes lost, over 200 simulated hours.
SimulatePythonMerge histograms vs. average p99
Ten servers with very different traffic: average of their p99s, the true p99, and the p99 from merged histogram buckets.
SimulatePython with NumPyCount your series
A prometheus_client endpoint: series per metric against label cardinalities, then one label with 10,000 values added.
MeasurePython with prometheus_clientHead vs. tail sampling
100,000 traces, 1% errors: error traces kept by 1% head sampling and by tail sampling, and the tail sampler's buffer at 1 s and 30 s traces.
SimulatePythonFind the false green in your own pipeline
An hourly job starts receiving empty files: job-status alert, freshness check and volume band, and how long each takes to notice.
SimulatePythonBase-rate simulation of alerting
A million checks with a 1-in-1,000 incident rate and a perfect-recall detector at four false-positive rates: pages and precision.
SimulatePython
Can I reason about it under questioning?
Interview questions
Short questions on observability for data systems, such as metrics, logs and traces, freshness and lag, alerting and error budgets, canaries and capacity, and safe data deploys, each with an answer you could give aloud and a link to the full explanation.
Scenarios
Longer problems on observability for data systems, of the kind asked in senior interviews: each one estimated with numbers, diagnosed, and answered with its trade-offs.
The tensions this mechanism takes a position on. Each recurs across mechanisms.
Exactness vs. approximation
For example: sketches, sampling, ANN.
Latency vs. throughput
For example: batching, group commit, micro-batch.
Space vs. time
For example: compression, indexing, materialization, caching.
Fail-fast vs. fail-degraded
Reject or continue when something is wrong.
Cost as a substitutable resource
Money buys position on every other axis; the only axis denominated in a different unit.
Ideas this mechanism meets that keep coming back, mechanism after mechanism.
Absence is not evidence
Absence of an event is not absence of a failure.
Costs are conserved
Costs are conserved, never removed.
Which way the error points
Variance costs more than volume
Variance costs more than volume.
Granularity mismatch
Granularity mismatch creates amplification.
Show all 10 principlesShow fewer
Independence is assumed
Independence is assumed everywhere and is true almost nowhere.
Masking moves failure
Masking a failure moves it from a channel with a schema to a channel without one.
Guarantees have scope
Atomicity is bounded by the shared commit substrate.
Fixed capacity meets growth
A fixed-capacity structure meeting growing cardinality breaks the design.
The recovery path is the normal path
The recovery path should be the normal path.