Skip to content
About
Main content

Reproduce a deadlock, then remove it by ordering locks

Break it on purposeNeeds Docker, Python

Problem

A deadlock needs two transactions that each hold a the other wants. Two transfers between the same two accounts in opposite directions do exactly that: each locks its source row first and then waits for the other's. The database detects the cycle after deadlock_timeout and one transaction. Locking every row a transaction needs in a fixed global order (for example by key) makes the cycle impossible.

Two sessions against a table wallets(owner text primary key, balance int) with rows anna and ben. Session 1 moves 30 from anna to ben, session 2 from ben to anna, each updating its source row first. Then change both sessions to lock both rows in a fixed order first (SELECT ... WHERE owner IN ('anna','ben') ORDER BY owner FOR UPDATE) and repeat. Then set deadlock_timeout = '100ms' and measure how long the victim waited before it was aborted. 20 trials each.

The question. How often do opposite-order transfers deadlock, how long does the victim wait, and what does a fixed lock order change?


Predict

With deadlock_timeout = 1s, how long does the victim wait before PostgreSQL aborts it?

Run it

  1. 1Install uv once. It fetches Python and the lab's packages for you. This lab also runs a real system, so you need Docker.

    Terminal
    curl -LsSf https://astral.sh/uv/install.sh | sh
  2. 2Get the script. It is the whole lab: read it, then download it.

    reproduce-a-deadlock-then-remove-it.py61 linesDownload
    View the code
    reproduce-a-deadlock-then-remove-it.py
    # /// script
    # requires-python = ">=3.10"
    # dependencies = ["psycopg[binary]"]
    # ///
    # Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
    # https://thedataengineering.com/lab-code/LICENSE.txt
    """Reproduce a deadlock, then remove it.
    
    Two PostgreSQL sessions move 30 between anna and ben in opposite orders and
    deadlock; one is aborted after deadlock_timeout. Then both lock the two rows
    in a fixed order first (SELECT ... ORDER BY owner FOR UPDATE) and no deadlock
    occurs. Repeated 20 times each, and with deadlock_timeout = 100ms, measuring
    how long the victim waited.
    Needs the PostgreSQL service from the compose file (port 55801).
    """
    import argparse, json, threading, time
    import psycopg
    
    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    p.add_argument("--dsn", default="postgresql://postgres:lab@localhost:55801/postgres", help="Connection string.")
    p.add_argument("--trials", type=int, default=20, help="Trials per variant.")
    p.add_argument("--json", help="Also write the results to this JSON file.")
    args = p.parse_args()
    
    setup = psycopg.connect(args.dsn, autocommit=True)
    setup.execute("DROP TABLE IF EXISTS wallets"); setup.execute("CREATE TABLE wallets (owner text PRIMARY KEY, balance int)")
    setup.execute("INSERT INTO wallets VALUES ('anna', 1000), ('ben', 1000)")
    def session(first, second, ordered, timeout, out, barrier):
        c = psycopg.connect(args.dsn); c.execute(f"SET deadlock_timeout = '{timeout}'")
        t = time.perf_counter()
        try:
            with c.transaction():
                if ordered: c.execute("SELECT * FROM wallets WHERE owner IN ('anna', 'ben') ORDER BY owner FOR UPDATE")
                c.execute("UPDATE wallets SET balance = balance - 30 WHERE owner = %s", (first,))
                if barrier: barrier.wait()                                      # both hold their first row before taking the second
                time.sleep(0.05)
                c.execute("UPDATE wallets SET balance = balance + 30 WHERE owner = %s", (second,))
            out.append(("ok", time.perf_counter() - t))
        except psycopg.errors.DeadlockDetected:
            out.append(("deadlock", time.perf_counter() - t))
        finally: c.close()
    rows = []
    for ordered, timeout in ((False, "1s"), (False, "100ms"), (True, "1s")):
        deadlocks, waits = 0, []
        for _ in range(args.trials):
            out = []; barrier = None if ordered else threading.Barrier(2, timeout=10)
            ts = [threading.Thread(target=session, args=("anna", "ben", ordered, timeout, out, barrier)),
                  threading.Thread(target=session, args=("ben", "anna", ordered, timeout, out, barrier))]
            for t in ts: t.start()
            for t in ts: t.join()
            victims = [w for r, w in out if r == "deadlock"]; deadlocks += bool(victims); waits += victims
        total = setup.execute("SELECT sum(balance) FROM wallets").fetchone()[0]
        rows.append({"locking": "fixed order (FOR UPDATE first)" if ordered else "opposite orders", "deadlock_timeout": timeout,
                     "trials": args.trials, "deadlocks": deadlocks, "victim_wait_ms": (sum(waits) / len(waits) * 1000) if waits else None, "total_balance": total})
    for r in rows: print(f"{r['locking']:<32} deadlock_timeout {r['deadlock_timeout']:<6} deadlocks {r['deadlocks']:>2}/{r['trials']}" +
                          (f"   victim waited {r['victim_wait_ms']:.0f} ms" if r["victim_wait_ms"] else "") + f"   total balance {r['total_balance']}")
    
    if args.json:
        with open(args.json, "w") as f:
            json.dump({"tables": {"variants": rows}}, f)
    
  3. 3Get the services it runs against, pinned to the versions we measured. Save it next to the script.

    reproduce-a-deadlock-then-remove-it.compose.yml13 linesDownload
    View the code
    reproduce-a-deadlock-then-remove-it.compose.yml
    # PostgreSQL for the reproduce-a-deadlock-then-remove-it lab.
    services:
      postgres:
        image: postgres:17.6
        command: ["postgres", "-c", "max_connections=200", "-c", "shared_buffers=256MB"]
        environment:
          POSTGRES_PASSWORD: lab
        ports: ["55801:5432"]
        healthcheck:
          test: ["CMD", "pg_isready", "-U", "postgres"]
          interval: 1s
          retries: 30
    
  4. 4Start the services, run the script with the defaults first, then change a knob from the table below and run it again. Remove the services when you are done.

    Terminal
    docker compose -f reproduce-a-deadlock-then-remove-it.compose.yml up -d --wait
    uv run reproduce-a-deadlock-then-remove-it.py
    uv run reproduce-a-deadlock-then-remove-it.py --help     # every knob
    docker compose -f reproduce-a-deadlock-then-remove-it.compose.yml down -v
Knobs
Knob · defaultWhat it changes
--dsnpostgresql://postgres:lab@localhost:55801/postgres Connection string.
--trials20 Trials per variant.

Our run

20 trials each

20 trials each
Lockingdeadlock_timeoutDeadlocksVictim waited (ms)Total balance after
opposite orders1s201,0702,000
opposite orders100ms201662,000
fixed order (FOR UPDATE first)1s0—2,000
Reference runApple M2 · 8 cores · 8 GB
Output · uv run reproduce-a-deadlock-then-remove-it.py
opposite orders                  deadlock_timeout 1s     deadlocks 20/20   victim waited 1068 ms   total balance 2000
opposite orders                  deadlock_timeout 100ms  deadlocks 20/20   victim waited 166 ms   total balance 2000
fixed order (FOR UPDATE first)   deadlock_timeout 1s     deadlocks  0/20   total balance 2000
macOS 26.6.2 · Python 3.13.3 · psycopg 3.3.6 · postgres:17.6 · Docker 28.5.1 · run 2026-10-20

What we found

In opposite order, 20 of 20 trials deadlocked, and the victim waited 1,070 ms before being aborted; with deadlock_timeout = 100ms, 166 ms. Locking both rows in a fixed order first, 0 trials deadlocked. Money was conserved throughout: the total stayed 2,000.


Read it

Deadlock detection is a safety net with a delay: every deadlock costs the victim the timeout plus a retry. Lowering the timeout shortens the wait and makes the detector run more often. A consistent lock order removes the cycle, so there is nothing to detect.