Transactions & Concurrency Control
What happens when many users change the same data at the same time, and how do we keep the result correct?
Who this mechanism is for: a developer who writes code, calls APIs and knows basic SQL, but has never studied how a database lets many users work on the same data at once. You do not need any database-internals knowledge. Every technical term is explained the first time it appears, and again in the glossary at the end.
The story we will follow
Tomas Berg is a backend engineer at Kestrel Tickets, a small company that sells concert and sports tickets online. Each customer also has a prepaid wallet: customers add money to it and pay for tickets from it. Tomas owns the order service. It reserves seats, charges wallets and records orders in one SQL database that we will call orders-db. His colleague Lena Okafor is the database engineer and the on-call lead. She is calm, she measures everything, and her favourite question is "what is the abort rate?"
Normally Kestrel handles about 800 orders per second. Next month the company sells tickets for the Hollowmoon Tour. The sale will reach 8,000 orders per second for a few minutes, and most buyers will fight for the same 200 "floor" seats out of 50,000. Across this mechanism Tomas will meet five problems, one at the start of each article. Each problem looks like a bug. Each one is actually a consequence of arithmetic or of a design choice that every database makes. At the end of each article we go back and solve his problem with the mechanisms of that article.
This is the main idea of this mechanism:
How this mechanism is organized
- 1
Why concurrency is different
Why does ten times more throughput create a hundred times more conflict, and how should we think about correctness when many users act at once?
CaseTen times the orders, a hundred times the retries
- 2
Locking
How can a database guarantee a correct order without knowing the future? What are locks, latches, two-phase locking and deadlocks?
CaseA deadlock between two friends paying each other
- 3
Multiversion and optimistic schemes
What changes if readers keep old versions of data, or if we only check for conflict at the end, or remove the waiting completely? How do we choose?
CaseA 3-hour report, and a table that tripled
- 4
Transaction mechanics
What really happens at BEGIN, COMMIT and ABORT? How is "all or nothing" built when no layer below can do many writes at once?
CaseA timeout on COMMIT: did the top-up happen?
- 5
Distributed transactions
What breaks when the data lives on several machines or services, and what do real systems do about it?
CaseCharged twice, or a seat with no e-ticket
After that: a summary, hands-on labs, interview questions and a glossary.
One request: do the small calculations here with a pencil. The central results (for example "ten times the throughput is a hundred times the conflict") are not opinions. They are arithmetic, and the arithmetic predicts what your system will do.
What do I do, and what does each option cost?
Choose a concurrency scheme
Concurrent transactions write to the same data: locking, optimistic, commutative operations, partitioning or deterministic execution? Measure the conflict rate, shorten transactions, then follow five questions to one scheme.
ChooseDesign a commit
Building or choosing the commit path of a store: name the one atomic write it rests on, derive the commit-rate ceiling, and decide what an abort and a retry will cost.
DesignChoose a distributed commit
One operation writes to several systems: make it all-or-nothing, or let each system commit on its own and handle failures another way? Six questions lead to one of six designs, each with its costs and checks.
Choose
Can I see it myself?
Conflict grows with the square of concurrency (uniform versus skewed keys)
L concurrent transactions touching k random keys out of N: conflicting pairs against L²k²/2N, uniform and Zipf-skewed.
SimulatePython with NumPyPhantoms and gap locks depend on the index
MySQL/InnoDB: SELECT ... FOR UPDATE on a price range, with and without an index on price; which inserts wait, and how many locks are held.
Real systemDocker, PythonReproduce a deadlock, then remove it by ordering locks
Two PostgreSQL sessions move money between the same two wallets in opposite orders, 20 times; then with a fixed lock order; then with deadlock_timeout = 100ms.
Break itDocker, PythonOne old snapshot pins the version space
Two minutes of random updates on a million-row PostgreSQL table, without and with an old Repeatable Read snapshot open: size and dead tuples.
Real systemDocker, PythonPessimistic versus optimistic crossover
16 workers doing read-modify-write on a PostgreSQL counter table of K rows, with SELECT FOR UPDATE and with a version column: throughput and goodput from K = 1,000,000 to 1.
Real systemDocker, PythonWrite skew under snapshot isolation, and its fix
Two on-call agents each check that two are on shift and each go off shift: REPEATABLE READ against SERIALIZABLE; then 50 clients and the retry rate.
Real systemDocker, PythonThe idempotency key: the unknown outcome made harmless
A top-up whose acknowledgment is lost, retried naively and with an idempotency key in SQLite; then a replay after the key has expired.
Break itPythonSavepoint cliff in Postgres (more than 64 subtransactions)
A PostgreSQL transaction holding 0 to 200 subtransactions open while pgbench reads the same rows: read throughput across the 64-subtransaction cache.
Real systemDocker, PythonContended compare-and-swap: n²T/2 and the value of jitter
n writers each need one successful conditional write on a shared pointer: total time and wasted attempts with immediate retry and with jittered backoff.
SimulatePythonAvailability of unanimity: 0.999ⁿ versus a majority
Atomic commit needs every participant; consensus needs a majority: a^n against the binomial majority, then 10 million simulated rounds with and without rack failures.
SimulatePython with NumPy
Can I reason about it under questioning?
Interview questions
Short questions on transactions and concurrency control, such as conflict arithmetic, locking, MVCC, optimistic schemes, commit mechanics and distributed transactions, each with an answer you could give aloud and a link to the full explanation.
Scenarios
Longer problems on transactions and concurrency control, of the kind asked in senior interviews: each one estimated with numbers, diagnosed, and answered with its trade-offs.
The tensions this mechanism takes a position on. Each recurs across mechanisms.
Ordering vs. parallelism
Global order costs serialization; parallelism costs order.
Mutability vs. immutability
In-place update vs. append-only + GC; buys reproducibility and cheap concurrency, pays in space and janitorial work.
Consistency vs. availability vs. latency
For example: quorums, replication modes.
Durability vs. acknowledgment latency
How much must be confirmed before you say "yes".
Ideas this mechanism meets that keep coming back, mechanism after mechanism.
Costs are conserved
Costs are conserved, never removed.
Variance costs more than volume
Variance costs more than volume.
Designate one write
Atomicity over n items is atomicity over one designated item.
Sound approximations
When exactness is too expensive, use a sound approximation.
Granularity mismatch
Granularity mismatch creates amplification.
Show all 14 principlesShow fewer
Independence is assumed
Independence is assumed everywhere and is true almost nowhere.
Little's Law
Concurrency = Throughput × Latency.
Queues explode near full load
Response time explodes as utilization nears 100%.
Randomization replaces coordination
Randomization can replace coordination.
Immutable storage grows indirection
An immutable substrate grows an indirection layer plus a collector.
The algebra decides
What can be skipped, pushed down, parallelized, merged or precomputed is determined by the algebra of the function, not the layout of the data.
Power laws and skew
Coordination cannot be made cheaper
Coordination is the only thing you cannot make cheaper.
Retention is paid by the live path