Ack before or after processing: loss versus duplicates

Break it on purposeNeeds Python

Problem

A consumer does two writes for every record: the effect (process it) and the progress (store the offset it has reached). A crash can land between them, and which one went first decides what is lost. Saving the offset first means a crash after it skips the record: at-most-once. Processing first means a crash after it repeats the record on restart: at-least-once. There is no order that gives neither, short of making the two writes one.

Write a consumer that reads numbered lines from a file and "processes" each by appending it to out.txt, and stores its progress (the offset) in offset.txt. Make two versions: A writes the offset first, then processes; B processes first, then writes the offset. Kill the process with kill -9 at random moments, restart it, and repeat until it finishes. Then count in out.txt.

The script runs the consumer as a child process and kills it with SIGKILL after a random 50–300 ms, again and again, until it completes all 10,000 records.

The question. After dozens of random kills, which version loses numbers and which duplicates them?

Predict

After dozens of random kills, which version loses numbers?

A: save the offset, then processB: process, then save the offsetBothNeither

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

ack-before-or-after-processing-loss.py61 linesDownload

View the code

ack-before-or-after-processing-loss.py

# /// script
# requires-python = ">=3.10"
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Ack before or after processing: loss or duplicates.

A consumer reads numbers 0..9999, "processes" each by appending it to a
file, and stores its offset. Version A saves the offset first; version B
processes first. The process is killed with SIGKILL at random moments and
restarted until it finishes; then the output is checked for missing and
duplicated numbers.
"""
import argparse, collections, json, os, random, signal, subprocess, sys, tempfile, time

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--records", type=int, default=10_000, help="Records to process.")
p.add_argument("--max-kill-delay", type=float, default=0.3, help="Largest random delay before each kill, in seconds.")
p.add_argument("--seed", type=int, default=0, help="Random seed.")
p.add_argument("--worker", help=argparse.SUPPRESS)
p.add_argument("--dir", help=argparse.SUPPRESS)
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()

if args.worker:                                            # the consumer process
    os.chdir(args.dir)
    def load():
        try: return int(open("offset.txt").read())
        except Exception: return 0
    def save(o):
        open("offset.txt.tmp", "w").write(str(o)); os.replace("offset.txt.tmp", "offset.txt")
    out = open("out.txt", "a")
    for n in range(load(), args.records):
        if args.worker == "ack_first":
            save(n + 1); out.write(f"{n}\n"); out.flush()
        else:
            out.write(f"{n}\n"); out.flush(); save(n + 1)
        time.sleep(random.random() / 2000)
    sys.exit(0)

random.seed(args.seed)
rows = []
for mode, label in (("ack_first", "A: save the offset, then process"), ("process_first", "B: process, then save the offset")):
    d = tempfile.mkdtemp(prefix=f"ack-{mode}-")
    kills = 0
    while True:
        proc = subprocess.Popen([sys.executable, __file__, "--worker", mode, "--dir", d, "--records", str(args.records)])
        try:
            proc.wait(timeout=random.uniform(0.05, args.max_kill_delay)); break     # finished
        except subprocess.TimeoutExpired:
            os.kill(proc.pid, signal.SIGKILL); proc.wait(); kills += 1
    nums = [int(x) for x in open(os.path.join(d, "out.txt")).read().split()]
    counts = collections.Counter(nums)
    missing = args.records - len(counts); dups = sum(c - 1 for c in counts.values() if c > 1)
    rows.append({"version": label, "kills": kills, "missing": missing, "duplicated": dups})
    print(f"{label:<36} killed {kills:>3} times: {missing:>3} numbers missing, {dups:>3} duplicated")

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"versions": rows}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal
```
uv run ack-before-or-after-processing-loss.py
uv run ack-before-or-after-processing-loss.py --help     # every knob
```

Knobs
Knob · default	What it changes
--records10,000	Records to process.
--max-kill-delay0.3	Largest random delay before each kill, in seconds.
--seed0	Random seed.

Our run

10,000 records, killed until done

10,000 records, killed until done
Version	Kills	Missing	Duplicated
A: save the offset, then process	34	4	0
B: process, then save the offset	39	0	15

Reference runApple M2 · 8 cores · 8 GB

Output · uv run ack-before-or-after-processing-loss.py

A: save the offset, then process     killed  34 times:   4 numbers missing,   0 duplicated
B: process, then save the offset     killed  39 times:   0 numbers missing,  15 duplicated

macOS 26.6.2 · Python 3.13.3 · run 2026-10-20

What we found

Version A (offset first) was killed 34 times and lost 4 numbers, with 0 duplicates. Version B (process first) was killed 39 times and lost 0, with 15 duplicates. Never the reverse, as expected. Most kills landed elsewhere in the loop, so each version's failures are a small fraction of its kills.

Read it

This is the whole ack-then-process table made visible.

The loss-or-duplication choice is simply which of two writes goes first. Since loss is silent and duplicates can be made harmless, almost every pipeline should process first and make the sink idempotent; the stronger answer is to commit the offset in the same transaction as the effect.