Whole-query restart against task retry

Simulate a modelNeeds Python

Problem

A distributed query runs as many tasks in parallel. When one task fails, the engine can restart the whole query or retry only the failed task. Restarting is simple and lets stages stream rows to each other in memory, but an attempt succeeds only when no task fails. Retrying a single task needs each stage's output written where it can be read again (a materialized edge), which costs I/O on every run. If failures are random and independent, the probability that n tasks of T hours each all survive a failure rate λ per task-hour is about e^(−nTλ), so the expected number of whole-query attempts is about e^(nTλ).

Check that expected whole-query attempts grow like e^(nTλ). Simulate a job of n tasks, each of duration T hours, run all in parallel, with an independent failure rate λ per task-hour.

Then change the simulation so that only the failed tasks are retried (each failed task is retried until it succeeds). Count the extra task-time spent.

You will measure the average number of whole-query attempts for nTλ from 0.5 to 4 against e^(nTλ), and the extra task-time spent when only the failed tasks are retried.

The question. How fast do whole-query attempts grow as nTλ rises, and how much extra work does task retry cost at the same failure rate?

Predict

At nTλ = 4, how much extra task-time does retrying only the failed tasks cost?

Under 1%1 to 10%10 to 100%As much as restarting

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

restart-vs-task-retry.py56 linesDownload

View the code

restart-vs-task-retry.py

# /// script
# requires-python = ">=3.10"
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Whole-query restart against task retry.

A job of n tasks, each of duration T hours, all in parallel, with an
independent failure rate lambda per task-hour. Whole-query restart reruns
everything on any failure; task retry reruns only the task that failed.
"""
import argparse, json, math, random

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--tasks", type=int, default=1000, help="n: tasks in the job.")
p.add_argument("--task-hours", type=float, default=0.1, help="T: duration of one task, in hours.")
p.add_argument("--exposures", type=lambda s: [float(v) for v in s.split(",")], default=[0.5, 1.0, 2.0, 3.0, 4.0],
               help="Values of nT*lambda to simulate, comma-separated.")
p.add_argument("--runs", type=int, default=200, help="Simulated jobs per value.")
p.add_argument("--seed", type=int, default=0, help="Random seed.")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()
random.seed(args.seed)


def attempts_until_success(n, T, lam):
    p_task_ok = math.exp(-lam * T)            # probability that one task does not fail
    tries = 0
    while True:
        tries += 1
        if all(random.random() < p_task_ok for _ in range(n)): return tries


def task_retry_extra(n, T, lam):
    """Task-hours lost when only failed tasks are retried (each until it succeeds)."""
    lost = 0.0
    for _ in range(n):
        while (fail_at := random.expovariate(lam)) < T:   # the attempt dies after fail_at hours
            lost += fail_at
    return lost


n, T = args.tasks, args.task_hours
rows = []
for x in args.exposures:
    lam = x / (n * T)
    attempts = sum(attempts_until_success(n, T, lam) for _ in range(args.runs)) / args.runs
    extra = sum(task_retry_extra(n, T, lam) for _ in range(args.runs)) / args.runs / (n * T)
    rows.append({"exposure": x, "simulated": attempts, "e^x": math.exp(x), "task_retry_extra_pct": extra * 100})
    print(f"nT*lam = {x:<4}  whole-query attempts: {attempts:6.2f}  (e^x = {math.exp(x):6.2f})   "
          f"task retry: +{extra:.2%} task-time")

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"attempts": rows}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal
```
uv run restart-vs-task-retry.py
uv run restart-vs-task-retry.py --help     # every knob
```

Knobs
Knob · default	What it changes
--tasks1000	n: tasks in the job.
--task-hours0.1	T: duration of one task, in hours.
--exposures0.5, 1.0, 2.0, 3.0, 4.0	Values of nT*lambda to simulate, comma-separated.
--runs200	Simulated jobs per value.
--seed0	Random seed.

Our run

Whole-query attempts until one succeedsApple M2 · 8 cores · 8 GB

Simulatede^(nTλ)

Data table

Whole-query attempts until one succeeds
nTλ	Simulated	e^(nTλ)
0.5	1.66	1.65
1	2.54	2.72
2	7.54	7.39
3	20	20.1
4	53.3	54.6

macOS 26.6.2 · Python 3.13.3 · run 2026-10-20

Task retry instead

Task retry instead
nTλ	Whole-query attempts	Task retry: extra task-time
0.5	1.66	0.025%
1	2.54	0.054%
2	7.54	0.104%
3	20	0.151%
4	53.3	0.199%

Output · uv run restart-vs-task-retry.py

nT*lam = 0.5   whole-query attempts:   1.66  (e^x =   1.65)   task retry: +0.02% task-time
nT*lam = 1.0   whole-query attempts:   2.54  (e^x =   2.72)   task retry: +0.05% task-time
nT*lam = 2.0   whole-query attempts:   7.54  (e^x =   7.39)   task retry: +0.10% task-time
nT*lam = 3.0   whole-query attempts:  20.02  (e^x =  20.09)   task retry: +0.15% task-time
nT*lam = 4.0   whole-query attempts:  53.30  (e^x =  54.60)   task retry: +0.20% task-time

What we found

The simulation tracks e^(nTλ) closely: 2.54 attempts at nTλ = 1 and 53.3 at nTλ = 4, where e⁴ = 54.6. The small gaps are simulation noise from averaging 200 jobs per point.

Retrying only the failed tasks cost 0.199% extra task-time at nTλ = 4, the point where whole-query restart needed about 53.3 complete runs. That is the chapter's contrast in numbers: a small percentage against a multiple of the whole job.

Read it

This single product (task count × duration × failure rate) is the dividing line between the pipelined MPP world (massively parallel engines that stream rows between stages and restart the whole query on failure) and the materialized DAG world (engines that write each stage's output and retry single tasks). It is why the question "should stage edges be materialized?" has different right answers for a 200 ms dashboard query and a six-hour batch job. These are not two philosophies. They are one inequality.

The cost of recovering from failure is not eliminated by either choice. It is relocated into I/O and latency (materialize) or into restart probability and memory residency (stream).

Go further

The cost on the other side (writing intermediates) you can estimate from your storage speed.