Scenarios
Longer problems, like those in senior system design interviews. A strong answer estimates with numbers first, names the mechanism, then recommends a fix and says what it costs.
1A flash-sale checkout runs at 8,000 orders per second, and 12% of transactions are retried after conflicts, although CPU is only at 55%. Normal traffic is 800 per second. Diagnose and fix.Show the answer
Estimate
If each transaction lasts W = 5 ms, then L = λW = 8,000 × 0.005 = 40 in flight, against 4 at 800 per second. Pairs of live transactions grow from about 8 to about 800, a hundredfold. And if 90% of buyers want the same 200 seats, the effective number of keys is closer to 200 than 50,000, so the conflict probability is far above a uniform estimate.
Mechanism
Optimistic or snapshot-based schemes pay for conflicts in wasted, retried work; locking would pay in queueing instead.
Fix, in order
- Measure the real abort rate under replayed traffic.
- Shorten W: move external calls out of the transaction. Its effect on conflicts is quadratic.
- Reformulate the hot operations so they commute: sub-counters or reservations per section.
- Partition by event or section so most orders stay local.
Trade-offs
Sub-counters need rebalancing, and may refuse a sale while stock sits in another sub-counter. Partitioning makes orders that span sections expensive.
Recommend
Shorten W and split the hot counter first; do not change the isolation scheme until the abort rate is known.
Read moreThe arithmetic that governs this mechanismUniform accessCommutativity
2Choose between locking and optimistic concurrency control for three workloads: a staff rota table with 20 writes per second; a wallet table with 5,000 transfers per second spread over millions of accounts; and a single inventory row decremented 3,000 times per second.Show the answer
The formula
Use the conflict probability p ≈ L² k² / 2N: L transactions in flight, each touching k keys, out of N effective keys.
Rota table
L is tiny, so p is almost zero: optimistic, or serializable snapshot isolation, which also catches write skew.
Wallets
With W = 3 ms, L = 15; with k = 2 and N = 10⁶, p ≈ 15² × 4 / (2 × 10⁶) ≈ 0.05%, well below the 1–2% break-even: optimistic wins, and its retries are rare. Watch for a few giant accounts, which shrink the effective N.
One inventory row
N ≈ 1, so any scheme only chooses between queueing and burning CPU. The answer is not a scheme but a reformulation: partitioned counters or escrow reservations, which commute.
3A nightly report sometimes fails with 'snapshot too old' on one database, and on another a table has tripled in size. Your CTO asks whether MVCC is broken.Show the answer
Answer
MVCC is working as designed. Old versions must be kept as long as the oldest live snapshot might need them, so a long read forces a choice of who pays.
The two databases
On the PostgreSQL-style system, versions live in the table, so writers pay in bloat, which is not recoverable without rewriting the table. On an Oracle-style system, versions live in a bounded undo area, so the reader pays with an error. (InnoDB keeps undo too, but its history grows instead of failing the reader.)
Remedies
Move long reads to a replica; split the report into shorter reads, or paginate by key; set idle-in-transaction and statement timeouts; and monitor the age of the oldest snapshot, bloat and undo retention before deploying.
The cost is conserved
Space, a collector, and the oldest reader's veto over both.
Read moreSpace and the oldest-reader problemVersion storage layout
4A business rule that spans rows, 'at least one agent must be on shift', is violated in production even though every change runs in a transaction. Why, and how do you fix it without making every query serializable?Show the answer
Diagnose
Write skew under snapshot isolation. Two transactions each read the same set of agents, each sees another agent on shift, and each takes a different agent off. Snapshot isolation checks only write–write conflicts, and these wrote different rows.
Options
- Run only this transaction at SERIALIZABLE (serializable snapshot isolation), which aborts one side of a dangerous dependency pattern; expect a small abort rate and add a retry loop.
- Lock the rows the check depends on (
SELECT … FOR UPDATEon all agents), at the price of queueing. - Make the conflict explicit by having both transactions update one shared guard row.
- Use a database constraint, if the rule can be expressed as one.
Cost
State it as an abort rate. How coarsely reads are tracked (row, page or table) sets how many aborts are false alarms.
5After a DBA dropped an index that nobody seemed to use, support queries started deadlocking. The SQL did not change. Explain, and prevent a recurrence.Show the answer
Explain
True predicate locking is intractable, so databases lock ranges of an index instead (next-key and gap locks). Which locks a statement takes therefore depends on the index the optimizer chose; without a suitable index, the only safe choice is to lock much more, up to the whole table. Dropping the index changed the plans, and with them the lock footprints and the deadlock structure. Isolation behaviour depends on the plan.
Prevent
Treat index and plan changes as concurrency changes, and test them under realistic concurrent load. Keep a deadlock log and a deadlock retry in the application, take locks in a consistent order, and keep transactions short.
Read morePredicate locks, and why nobody implements themAvoiding deadlock by ordering
6Design deadlock handling for a database sharded across three regions.Show the answer
Rule out detection
It would need a global waits-for graph built from a consistent distributed snapshot, which is slow and stale.
Prevent
Timestamp-based wound-wait, as Spanner does. Older transactions abort younger lock holders, which wastes little work and lets short transactions wait out brief locks. A restarted transaction keeps its original timestamp, so it cannot starve. Every decision is a local comparison, with no communication.
Backstop
Lock-wait timeouts only as a safety net; no correct value exists, because waits have a heavy tail.
Cost
Some unnecessary aborts: prevention aborts whenever a cycle might form, not only when one has.
Read morePrevention by timestampWhy distributed systems prevent instead of detect
7A team proposes replacing PostgreSQL with a system that runs one thread per partition, to avoid locks entirely. Evaluate the proposal.Show the answer
The argument for it
With the working set in memory, a transaction takes about 50 µs, so one core handles about 20,000 transactions per second per partition, and with one transaction at a time no concurrency control is needed.
Check
- Does the partition key match the transaction boundary? Each cross-partition transaction stalls the thread for a coordination round trip, so throughput falls off a cliff as their share grows.
- Does the working set stay in memory? When disk I/O returns, transactions get long and the argument reverses.
- Can every transaction be a one-shot, non-interactive procedure?
Recommend
Adopt it only for workloads with a clean partition key, such as per-user or per-event state, and keep the rest in a general-purpose database.
Cost
Concurrency control has become a data-modelling decision you must defend permanently.
Read moreSingle-threaded per partitionThe boundary condition
8A mobile app calls your top-up API, and on a bad network the response is lost. How do you make the API safe, and what are the limits?Show the answer
Name the problem
The client cannot tell whether the top-up committed: the outcome is unknown.
Design
The client generates one idempotency key per intent and sends it on every retry. The server inserts the key into a unique column inside the same transaction as the balance change. A duplicate key returns the stored result instead of applying the change again.
Limits
The key table needs a retention window, say 7 days. A retry after a key is evicted would be applied twice, so cap the client's retry period below the retention. Deduplication state also needs clean-up and monitoring.
Safety net
A reconciliation report against the payment provider, because the data pays quietly whenever the window assumption breaks.
9Six Spark jobs write to one Iceberg table, and 30% of their commits fail, costing hours of work. Diagnose and fix.Show the answer
Diagnose
The unit of conflict is a file, not a row. If each job touches about 100 of 10,000 files, any two jobs collide with probability near 1, and a commit taking about 50 ms makes each window of exposure long.
Check
Whether failed commits can be rebased: an append can simply retry its metadata step, but a MERGE or a compaction that rewrote files others read cannot.
Fix
One writer per table, or disjoint partitions per writer. Run compaction in a window without ingestion. Use append mode where possible. Retry with jittered backoff and a retry budget, and commit in fewer, larger batches: the ceiling is tens of commits per second.
Trade-off
Partitioning writers constrains the data model.
Read moreOptimistic commit on a metadata logWrite conflicts and retry semantics
10The CTO wants two-phase commit across the order database, the inventory database and an external payment API. Evaluate the idea.Show the answer
Capability check
The payment API has no prepare phase, so two-phase commit is not available across all three.
The numbers
For the two databases, locks are held for the local work plus two round trips and two log flushes, and with hot rows that duration enters the conflict rate squared. Availability is the product of all participants, 0.999³ ≈ 0.997.
Recommend
A saga: reserve inventory, authorize the payment, then capture it, putting the irreversible step last, with compensating actions that are idempotent. Publish events through an outbox, with idempotent consumers.
Failure modes
Intermediate states are visible (use a pending status), compensations can get stuck (route them to a human queue), and you need a reconciliation job.
Read moreSagasOrder the irreversible step lastWhat distributed transactions forbid
11Design the commit path for a database that must never lose an acknowledged commit, at 20,000 commits per second, on an SSD whose fsync takes 50 µs.Show the answer
The order
Write ahead and force at commit: the commit is the moment its record becomes durable. Release locks after the flush, and acknowledge last.
Estimate
Without batching, 1 ÷ 50 µs = 20,000 flushes per second, exactly the target, with no headroom. Use group commit: its self-sizing batches let throughput rise with load.
An explicit option
For tables that can tolerate a tiny loss window, offer asynchronous commit with a bounded window, as an explicit trade.
Clients
They still face the unknown-outcome window, so their retries need idempotency keys.
12Events from an order database must reach 12 consumer teams. Choose between a transactional outbox and raw change data capture of the database's log.Show the answer
What they share
Both make the database commit the atomic event, and both deliver at least once, so consumers need idempotence either way.
Outbox
You design event shapes independently of your schema, at the price of a table with constant insert-and-delete churn (index bloat, vacuum pressure), eased by reading it through CDC rather than polling.
Raw CDC
No extra table, but 12 teams become coupled to your physical schema: renaming a column breaks them.
Recommend
With 12 consumers and long-lived contracts, the outbox. Keep per-key order by partitioning events by order ID.
Read moreTransactional outbox
13A global database must provide external consistency. Compare commit wait with uncertainty restarts for a workload that is 95% reads.Show the answer
The same cost, billed differently
Both pay for the same clock uncertainty ε. Commit wait (TrueTime-style) puts ε on every write as pure latency, and reads stay cheap. Hybrid logical clocks with an uncertainty interval put it on reads, as restarts whose rate grows with clock skew.
Decide
With 95% reads, billing the 5% of writes is usually cheaper.
Also weigh
The cost of making ε small (GPS and atomic clocks, or a cloud bounded-time service) against the latency it saves. And correct the folklore: atomic clocks are not required, only a bound on uncertainty.
Read moreBounded clock uncertainty
14Your two-phase-commit coordinator crashed in production, and 3,000 rows are locked by prepared transactions. What now, and how do you stop this becoming an incident next time?Show the answer
Now
Recover the coordinator first: its log holds the decisions, and prepared participants cannot decide alone. If recovery is impossible, an operator makes heuristic decisions, accepting a possible atomicity violation, and reconciles by hand.
Prevent
Replicate the coordinator with consensus; set timeouts and alerts on prepared transactions; keep lock-hold times short by repartitioning so most transactions are local; and design the reconciliation path before launch.
Know the default
Under presumed abort, a transaction with no decision record is treated as aborted.
15A service writes to PostgreSQL and publishes to Kafka. After a restart, consumers see duplicate events. Is exactly-once possible here?Show the answer
Answer
Exactly-once delivery is not; exactly-once processing is.
Design
Use an outbox so the database commit covers the intent to publish. Make consumers idempotent, by a natural key or a deduplication table whose retention exceeds how far back the broker can replay. For Kafka-to-Kafka stages, transactions commit the output and the input offset as one atomic event.
Watch
Deduplication entries expiring is the silent failure; monitor with reconciliation counts.