Head vs. tail sampling

Simulate a modelNeeds Python

Problem

Head sampling decides whether to keep a trace when it starts, before anyone knows how it ends, so it keeps the same share of error traces as of everything else. Tail sampling waits until the trace is complete and keeps every error, but it must hold every span of every trace in flight until the decision: its memory is the arrival rate times the trace duration times the spans per trace.

Generate 100,000 synthetic traces where 1% are errors. Apply (a) head sampling at 1% and (b) tail sampling that keeps all error traces and 1% of the others. Count the error traces kept. Then measure the peak number of buffered spans for the tail sampler, at 1,000 traces per second of 20 spans, with trace durations of 1 s and 30 s, and compare it with rate × duration.

The question. How many error traces does each sampler keep, and how much does the tail sampler have to buffer?

Predict

For 30 s traces at 1,000 traces a second (20 spans each), how many spans does the tail sampler buffer at peak?

Tens of thousandsHundreds of thousandsMillions

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

head-vs-tail-sampling.py49 linesDownload

View the code

head-vs-tail-sampling.py

# /// script
# requires-python = ">=3.10"
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Head against tail sampling.

100,000 synthetic traces, 1% of them errors. Head sampling decides at the
start (keep 1%, before anyone knows how the trace ends); tail sampling waits
for the trace to finish and keeps every error plus 1% of the rest. Counts the
error traces each keeps, then simulates the tail sampler's buffer: traces
arrive at a fixed rate and are held until complete, with durations of 1 s and
30 s, and the peak number of buffered spans is compared with rate × duration.
"""
import argparse, heapq, json, random

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--traces", type=int, default=100_000, help="Traces.")
p.add_argument("--error-share", type=float, default=0.01, help="Share of traces that are errors.")
p.add_argument("--rate", type=float, default=1000, help="Traces per second (buffer simulation).")
p.add_argument("--spans", type=int, default=20, help="Spans per trace.")
p.add_argument("--seed", type=int, default=0, help="Random seed.")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()
random.seed(args.seed)

errors = [random.random() < args.error_share for _ in range(args.traces)]
head = sum(e for e in errors if random.random() < 0.01)
tail = sum(errors)                                                      # every error is kept
tail_total = tail + sum(1 for e in errors if not e and random.random() < 0.01)
sampling = [{"method": "head sampling, 1%", "errors_total": sum(errors), "errors_kept": head},
            {"method": "tail sampling, all errors + 1%", "errors_total": sum(errors), "errors_kept": tail, "traces_kept": tail_total}]

buffers = []
for duration in (1, 30):
    ends, buffered, peak, t = [], 0, 0, 0.0
    for i in range(int(args.rate * 120)):                               # two minutes of arrivals
        t = i / args.rate
        while ends and ends[0] <= t: heapq.heappop(ends); buffered -= args.spans
        d = random.uniform(0.5, 1.5) * duration                         # durations around the nominal value
        heapq.heappush(ends, t + d); buffered += args.spans; peak = max(peak, buffered)
    buffers.append({"duration_s": duration, "peak_spans": peak, "rate_x_duration": args.rate * duration * args.spans})
for r in sampling: print(f"{r['method']:<32} kept {r['errors_kept']:>5} of {r['errors_total']} error traces")
for r in buffers: print(f"tail buffer, traces lasting ~{r['duration_s']:>2} s: peak {r['peak_spans']:,} spans buffered (rate × duration × spans = {r['rate_x_duration']:,.0f})")

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"sampling": sampling, "buffers": buffers}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal
```
uv run head-vs-tail-sampling.py
uv run head-vs-tail-sampling.py --help     # every knob
```

Knobs
Knob · default	What it changes
--traces100,000	Traces.
--error-share0.01	Share of traces that are errors.
--rate1000	Traces per second (buffer simulation).
--spans20	Spans per trace.
--seed0	Random seed.

Our run

Error traces kept

Error traces kept
Method	Error traces	Kept
head sampling, 1%	991	10
tail sampling, all errors + 1%	991	991

Tail sampler buffer

Tail sampler buffer
Trace duration (s)	Peak spans buffered	Rate × duration × spans
1	20.8K	20K
30	604K	600K

Reference runApple M2 · 8 cores · 8 GB

Output · uv run head-vs-tail-sampling.py

head sampling, 1%                kept    10 of 991 error traces
tail sampling, all errors + 1%   kept   991 of 991 error traces
tail buffer, traces lasting ~ 1 s: peak 20,840 spans buffered (rate × duration × spans = 20,000)
tail buffer, traces lasting ~30 s: peak 603,840 spans buffered (rate × duration × spans = 600,000)

macOS 26.6.2 · Python 3.13.3 · run 2026-10-20

What we found

Head sampling kept 10 of 991 error traces; tail sampling kept all 991. The tail sampler's buffer peaked at 20.8K spans for 1 s traces and 604K for 30 s traces, within a few percent of rate × duration × spans: thirty times the duration, thirty times the memory.

Read it

Tail sampling buys the traces worth keeping with memory proportional to trace duration, and the longest traces set the bill. Systems with long-running traces cap the wait (decide after a timeout, keeping what is known) or sample in two stages, head-sampling the boring majority and tail-sampling the rest.