Distributed tracing
Why must causality be carried in-band, and when should a trace be sampled?
A trace follows one request through many services. It is made of spans: one span is one unit of work (a function call, a database query) with a start time, duration and a link to its parent span.
The constraint is informational and hard: causality cannot be reconstructed from independent per-node observations. Why?
- Clocks on different machines disagree, so timestamps cannot order events across machines.
- Concurrent requests interleave, so lines from two requests look the same.
Therefore an identifier must travel in-band with the work: it is passed along in headers or message metadata at every hop. (For example, an HTTP header such as traceparent: 00-<trace-id>-<span-id>-01.) The consequence is severe: one uninstrumented hop cuts the graph. Tracing is all-or-nothing in a way metrics never are. A metric from one service is useful alone. A trace through nine of ten services is cut at the tenth. This explains how tracing gets adopted (slowly, then all at once) better than the maturity of the tools does.
Sampling: deciding before or after
Keeping every trace is too expensive, so sampling is forced by cost. The choice is a clean early vs. late binding trade:
| Head-based sampling | Tail-based sampling | |
|---|---|---|
| When it decides | at the start (ingress) of the request | after the whole trace is assembled |
| Cost | cheap and stateless | must buffer every span until the trace finishes |
| Weakness | decides before knowing if the trace is interesting: with 1% sampling you keep 1% of boring traces and drop 99% of the rare errors | memory = span rate × trace duration; spans of one trace arrive at different collectors, so the collectors must coordinate |
| Strength | simple | keeps what matters (errors, slow traces) |
Exemplars: the cheap join
An exemplar is a trace id attached to a histogram bucket. The aggregate carries a pointer into the identity-preserving signal. It is a join key between signal types. This turns "show me a slow one" into a two-click operation (click the slow bucket on the latency chart, open an example trace) instead of a search through millions of traces.
Where tracing breaks down for data pipelines
A span tree assumes a request whose life is shorter than human patience. A record's path through a pipeline lasts days, spreads across many workers in shuffles (redistributions of data between workers), and merges in joins. That shape is a DAG with fan-in (a graph where many parents feed one child), which the "one parent per span" model cannot express.
The honest statement: record-level provenance at production scale is unsolved. What works is to sample individual records through the pipeline and accept coverage in proportion to the sample rate. Above that granularity, dataset-level lineage takes over: a record of which tables and jobs produced which other tables.
One line: causality must be carried, not inferred, and you choose whether to decide what to keep before or after you know what happened.