Fencing tokens versus a paused leader (about 30 minutes)
Break it on purposeNeeds Python
Problem
A leader is chosen for an epoch and holds a lease: a promise that nobody else will lead until it expires. But a leader can pause (a long garbage-collection pause, a stalled VM) past its lease without knowing it. Meanwhile a new leader is elected for a higher epoch and starts writing. When the old leader wakes up, it still believes it leads, and writes. A lease alone cannot stop that write; a fence can: the resource remembers the highest epoch it has seen and rejects anything lower.
Claim to test: a lease alone leaks at the boundary, and a fence checked by the resource closes the gap (What coordination forbids, item 3).
Write a small Python program with a "seat store" (a dictionary) and two leaders. Leader A gets epoch 1, then sleeps for 3 seconds (this simulates a GC pause). Meanwhile leader B gets epoch 2 and writes. Then A wakes up and writes.
The question. When a paused leader writes after its lease expired, who owns the seat at the end, with and without a fence, and how often does it go wrong over random timings?
Predict
With fenced=False both writes return OK and the last writer (A, the stale leader) overwrites B. With fenced=True A's write is REJECTED and B's stays.
Run it
1Install uv once. It fetches Python and the lab's packages for you.
Terminal curl -LsSf https://astral.sh/uv/install.sh | sh2Get the script. It is the whole lab: read it, then download it.
View the codeHide the code
fencing-tokens-versus-a-paused-leader.py # /// script # requires-python = ">=3.10" # /// # Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT # https://thedataengineering.com/lab-code/LICENSE.txt """Fencing tokens versus a paused leader. Leader A holds epoch 1 and pauses (a GC pause) past its lease; leader B gets epoch 2 and writes; A wakes up and writes. The seat store accepts any write, or rejects writes with an epoch lower than the highest it has seen. Then 1,000 runs with random pauses. """ import argparse, json, random, threading, time p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) p.add_argument("--pause", type=float, default=3.0, help="Seconds leader A is paused.") p.add_argument("--b-at", type=float, default=1.2, help="Seconds after start that leader B writes.") p.add_argument("--runs", type=int, default=1000, help="Runs with random pauses (scaled 100x faster).") p.add_argument("--seed", type=int, default=0, help="Random seed.") p.add_argument("--json", help="Also write the results to this JSON file.") args = p.parse_args() random.seed(args.seed) def once(fenced, pause_a, b_at): store, state, lock, out = {}, {"highest": 0}, threading.Lock(), [] def write(owner, epoch): with lock: if fenced and epoch < state["highest"]: return "REJECTED" state["highest"] = max(state["highest"], epoch); store["A12"] = (owner, epoch); return "OK" def a(): time.sleep(pause_a); out.append(("A", write("A", 1))) def b(): time.sleep(b_at); out.append(("B", write("B", 2))) ts = [threading.Thread(target=a), threading.Thread(target=b)] [t.start() for t in ts]; [t.join() for t in ts] return out, store["A12"][0] rows = [] for pause in (args.pause, 0.5): for fenced in (False, True): out, owner = once(fenced, pause, args.b_at) rows.append({"pause_s": pause, "fenced": "yes" if fenced else "no", "writes": ", ".join(f"{w} {r}" for w, r in out), "final_owner": owner}) print(f"A paused {pause:g} s, fenced = {fenced!s:<5} {rows[-1]['writes']:<24} final owner {owner}") wrong = {False: 0, True: 0} for _ in range(args.runs): pause, b_at = random.uniform(0, 4) / 100, random.uniform(0, 4) / 100 # the same timings, 100x faster for fenced in (False, True): _, owner = once(fenced, pause, b_at) wrong[fenced] += owner == "A" and pause > b_at # A wrote after B: a stale leader won print(f"{args.runs} random runs: stale leader's write kept in {wrong[False]} unfenced, {wrong[True]} fenced") if args.json: with open(args.json, "w") as f: json.dump({"tables": {"cases": rows, "summary": [{"runs": args.runs, "wrong_unfenced": wrong[False], "wrong_fenced": wrong[True]}]}}, f)3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal uv run fencing-tokens-versus-a-paused-leader.py uv run fencing-tokens-versus-a-paused-leader.py --help # every knob
| Knob · default | What it changes |
|---|---|
| --pause3 | Seconds leader A is paused. |
| --b-at1.2 | Seconds after start that leader B writes. |
| --runs1000 | Runs with random pauses (scaled 100x faster). |
| --seed0 | Random seed. |
Our run
One run each
| A paused (s) | Fenced | Writes | Final owner |
|---|---|---|---|
| 3 | no | B OK, A OK | A |
| 3 | yes | B OK, A REJECTED | B |
| 0.5 | no | A OK, B OK | B |
| 0.5 | yes | A OK, B OK | B |
A paused 3 s, fenced = False B OK, A OK final owner A
A paused 3 s, fenced = True B OK, A REJECTED final owner B
A paused 0.5 s, fenced = False A OK, B OK final owner B
A paused 0.5 s, fenced = True A OK, B OK final owner B
1000 random runs: stale leader's write kept in 484 unfenced, 0 fencedWhat we found
With a 3-second pause, the unfenced store ended with the stale leader A as owner (A); with the fence, A's write was rejected and B kept the seat (B). With a 0.5-second pause, A wrote before B and both modes ended the same way, as expected.
Over 1,000 runs with random pauses, the stale leader's write was kept in 484 runs without the fence, every run in which A's pause outlasted B's start, and in 0 with it.
Read it
Now change the sleep in A to 0.5 s: the result is the same in both modes, which shows that the fence only matters in the bad timing case. Count how many of 1,000 runs with random sleeps produce a wrong final owner in unfenced mode.
That is why the bug survives testing: it needs a pause longer than the lease, which is rare on a quiet test machine and routine under production load. The fence works because it does not depend on timing at all: the resource compares two integers, and the comparison is right whatever the clocks did.