Skip to content
About
Main content

Fencing tokens versus a paused leader (about 30 minutes)

Break it on purposeNeeds Python

Problem

A is chosen for an and holds a : a promise that nobody else will lead until it expires. But a leader can pause (a long garbage-collection pause, a stalled VM) past its lease without knowing it. Meanwhile a new leader is elected for a higher epoch and starts writing. When the old leader wakes up, it still believes it leads, and writes. A lease alone cannot stop that write; a fence can: the resource remembers the highest epoch it has seen and rejects anything lower.

Claim to test: a lease alone leaks at the boundary, and a fence checked by the resource closes the gap (What coordination forbids, item 3).

Write a small Python program with a "seat store" (a dictionary) and two leaders. Leader A gets epoch 1, then sleeps for 3 seconds (this simulates a GC pause). Meanwhile leader B gets epoch 2 and writes. Then A wakes up and writes.

The question. When a paused leader writes after its lease expired, who owns the seat at the end, with and without a fence, and how often does it go wrong over random timings?


Predict

Over 1,000 runs with random pauses and no fence, in how many does the stale leader's write survive?

Run it

  1. 1Install uv once. It fetches Python and the lab's packages for you.

    Terminal
    curl -LsSf https://astral.sh/uv/install.sh | sh
  2. 2Get the script. It is the whole lab: read it, then download it.

    fencing-tokens-versus-a-paused-leader.py56 linesDownload
    View the code
    fencing-tokens-versus-a-paused-leader.py
    # /// script
    # requires-python = ">=3.10"
    # ///
    # Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
    # https://thedataengineering.com/lab-code/LICENSE.txt
    """Fencing tokens versus a paused leader.
    
    Leader A holds epoch 1 and pauses (a GC pause) past its lease; leader B gets
    epoch 2 and writes; A wakes up and writes. The seat store accepts any write,
    or rejects writes with an epoch lower than the highest it has seen. Then 1,000
    runs with random pauses.
    """
    import argparse, json, random, threading, time
    
    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    p.add_argument("--pause", type=float, default=3.0, help="Seconds leader A is paused.")
    p.add_argument("--b-at", type=float, default=1.2, help="Seconds after start that leader B writes.")
    p.add_argument("--runs", type=int, default=1000, help="Runs with random pauses (scaled 100x faster).")
    p.add_argument("--seed", type=int, default=0, help="Random seed.")
    p.add_argument("--json", help="Also write the results to this JSON file.")
    args = p.parse_args()
    random.seed(args.seed)
    
    
    def once(fenced, pause_a, b_at):
        store, state, lock, out = {}, {"highest": 0}, threading.Lock(), []
        def write(owner, epoch):
            with lock:
                if fenced and epoch < state["highest"]: return "REJECTED"
                state["highest"] = max(state["highest"], epoch); store["A12"] = (owner, epoch); return "OK"
        def a(): time.sleep(pause_a); out.append(("A", write("A", 1)))
        def b(): time.sleep(b_at); out.append(("B", write("B", 2)))
        ts = [threading.Thread(target=a), threading.Thread(target=b)]
        [t.start() for t in ts]; [t.join() for t in ts]
        return out, store["A12"][0]
    
    
    rows = []
    for pause in (args.pause, 0.5):
        for fenced in (False, True):
            out, owner = once(fenced, pause, args.b_at)
            rows.append({"pause_s": pause, "fenced": "yes" if fenced else "no", "writes": ", ".join(f"{w} {r}" for w, r in out), "final_owner": owner})
            print(f"A paused {pause:g} s, fenced = {fenced!s:<5}  {rows[-1]['writes']:<24} final owner {owner}")
    
    wrong = {False: 0, True: 0}
    for _ in range(args.runs):
        pause, b_at = random.uniform(0, 4) / 100, random.uniform(0, 4) / 100      # the same timings, 100x faster
        for fenced in (False, True):
            _, owner = once(fenced, pause, b_at)
            wrong[fenced] += owner == "A" and pause > b_at                        # A wrote after B: a stale leader won
    print(f"{args.runs} random runs: stale leader's write kept in {wrong[False]} unfenced, {wrong[True]} fenced")
    
    if args.json:
        with open(args.json, "w") as f:
            json.dump({"tables": {"cases": rows, "summary": [{"runs": args.runs, "wrong_unfenced": wrong[False], "wrong_fenced": wrong[True]}]}}, f)
    
  3. 3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.

    Terminal
    uv run fencing-tokens-versus-a-paused-leader.py
    uv run fencing-tokens-versus-a-paused-leader.py --help     # every knob
Knobs
Knob · defaultWhat it changes
--pause3 Seconds leader A is paused.
--b-at1.2 Seconds after start that leader B writes.
--runs1000 Runs with random pauses (scaled 100x faster).
--seed0 Random seed.

Our run

One run each

One run each
A paused (s)FencedWritesFinal owner
3noB OK, A OKA
3yesB OK, A REJECTEDB
0.5noA OK, B OKB
0.5yesA OK, B OKB
Reference runApple M2 · 8 cores · 8 GB
Output · uv run fencing-tokens-versus-a-paused-leader.py
A paused 3 s, fenced = False  B OK, A OK               final owner A
A paused 3 s, fenced = True   B OK, A REJECTED         final owner B
A paused 0.5 s, fenced = False  A OK, B OK               final owner B
A paused 0.5 s, fenced = True   A OK, B OK               final owner B
1000 random runs: stale leader's write kept in 484 unfenced, 0 fenced
macOS 26.6.2 · Python 3.13.3 · run 2026-10-20

What we found

With a 3-second pause, the unfenced store ended with the stale leader A as owner (A); with the fence, A's write was rejected and B kept the seat (B). With a 0.5-second pause, A wrote before B and both modes ended the same way, as expected.

Over 1,000 runs with random pauses, the stale leader's write was kept in 484 runs without the fence, every run in which A's pause outlasted B's start, and in 0 with it.


Read it

Now change the sleep in A to 0.5 s: the result is the same in both modes, which shows that the fence only matters in the bad timing case. Count how many of 1,000 runs with random sleeps produce a wrong final owner in unfenced mode.

That is why the bug survives testing: it needs a pause longer than the lease, which is rare on a quiet test machine and routine under production load. The fence works because it does not depend on timing at all: the resource compares two integers, and the comparison is right whatever the clocks did.