Skip to content
About
Main content

Scenarios

Longer problems, like those in senior system design interviews. A strong answer estimates with numbers first, names the mechanism, then recommends a fix and says what it costs.

Try each one before you open the answer.
1A flash-sale checkout runs at 8,000 orders per second, and 12% of transactions are retried after conflicts, although CPU is only at 55%. Normal traffic is 800 per second. Diagnose and fix.Show the answer

Estimate

If each transaction lasts W = 5 ms, then = 8,000 × 0.005 = 40 in flight, against 4 at 800 per second. Pairs of live transactions grow from about 8 to about 800, a hundredfold. And if 90% of buyers want the same 200 seats, the effective number of keys is closer to 200 than 50,000, so the conflict probability is far above a uniform estimate.

Mechanism

Optimistic or snapshot-based schemes pay for conflicts in wasted, retried work; locking would pay in queueing instead.

Fix, in order

  1. Measure the real rate under replayed traffic.
  2. Shorten W: move external calls out of the transaction. Its effect on conflicts is quadratic.
  3. Reformulate the hot operations so they : sub-counters or reservations per section.
  4. Partition by event or section so most orders stay local.

Trade-offs

Sub-counters need rebalancing, and may refuse a sale while stock sits in another sub-counter. Partitioning makes orders that span sections expensive.

Recommend

Shorten W and split the hot counter first; do not change the isolation scheme until the abort rate is known.

Read moreThe arithmetic that governs this mechanismUniform accessCommutativity

2Choose between locking and optimistic concurrency control for three workloads: a staff rota table with 20 writes per second; a wallet table with 5,000 transfers per second spread over millions of accounts; and a single inventory row decremented 3,000 times per second.Show the answer

The formula

Use the : L transactions in flight, each touching k keys, out of N effective keys.

Rota table

L is tiny, so p is almost zero: optimistic, or serializable isolation, which also catches .

Wallets

With W = 3 ms, L = 15; with k = 2 and N = 10⁶, p ≈ 15² × 4 / (2 × 10⁶) ≈ 0.05%, well below the 1–2% break-even: optimistic wins, and its retries are rare. Watch for a few giant accounts, which shrink the .

One inventory row

N ≈ 1, so any scheme only chooses between queueing and burning CPU. The answer is not a scheme but a reformulation: partitioned counters or escrow reservations, which commute.

Read moreOptimistic against pessimisticCommutativity

3A nightly report sometimes fails with 'snapshot too old' on one database, and on another a table has tripled in size. Your CTO asks whether MVCC is broken.Show the answer

Answer

is working as designed. Old versions must be kept as long as the oldest live snapshot might need them, so a long read forces a choice of who pays.

The two databases

On the PostgreSQL-style system, versions live in the table, so writers pay in bloat, which is not recoverable without rewriting the table. On an Oracle-style system, versions live in a bounded undo area, so the reader pays with an error. (InnoDB keeps undo too, but its history grows instead of failing the reader.)

Remedies

Move long reads to a replica; split the report into shorter reads, or paginate by key; set idle-in-transaction and statement timeouts; and monitor the age of the oldest snapshot, bloat and undo retention before deploying.

The cost is conserved

Space, a collector, and the oldest reader's veto over both.

Read moreSpace and the oldest-reader problemVersion storage layout

4A business rule that spans rows, 'at least one agent must be on shift', is violated in production even though every change runs in a transaction. Why, and how do you fix it without making every query serializable?Show the answer

Diagnose

Write skew under snapshot isolation. Two transactions each read the same set of agents, each sees another agent on shift, and each takes a different agent off. Snapshot isolation checks only write–write conflicts, and these wrote different rows.

Options

  1. Run only this transaction at SERIALIZABLE (serializable snapshot isolation), which aborts one side of a dangerous dependency pattern; expect a small abort rate and add a retry loop.
  2. the rows the check depends on (SELECT … FOR UPDATE on all agents), at the price of queueing.
  3. Make the conflict explicit by having both transactions update one shared guard row.
  4. Use a database constraint, if the rule can be expressed as one.

Cost

State it as an abort rate. How coarsely reads are tracked (row, page or table) sets how many aborts are false alarms.

Read moreThe anomaly SSI fixesThe honest cost of SSI

5After a DBA dropped an index that nobody seemed to use, support queries started deadlocking. The SQL did not change. Explain, and prevent a recurrence.Show the answer

Explain

True predicate locking is intractable, so databases lock ranges of an index instead (next-key and ). Which locks a statement takes therefore depends on the index the optimizer chose; without a suitable index, the only safe choice is to lock much more, up to the whole table. Dropping the index changed the plans, and with them the lock footprints and the deadlock structure. Isolation behaviour depends on the plan.

Prevent

Treat index and plan changes as concurrency changes, and test them under realistic concurrent load. Keep a deadlock log and a deadlock retry in the application, take locks in a consistent order, and keep transactions short.

Read morePredicate locks, and why nobody implements themAvoiding deadlock by ordering

6Design deadlock handling for a database sharded across three regions.Show the answer

Rule out detection

It would need a global built from a consistent distributed snapshot, which is slow and stale.

Prevent

Timestamp-based , as Spanner does. Older transactions abort younger lock holders, which wastes little work and lets short transactions wait out brief locks. A restarted transaction keeps its original timestamp, so it cannot starve. Every decision is a local comparison, with no communication.

Backstop

Lock-wait timeouts only as a safety net; no correct value exists, because waits have a .

Cost

Some unnecessary aborts: prevention aborts whenever a cycle might form, not only when one has.

Read morePrevention by timestampWhy distributed systems prevent instead of detect

7A team proposes replacing PostgreSQL with a system that runs one thread per partition, to avoid locks entirely. Evaluate the proposal.Show the answer

The argument for it

With the working set in memory, a transaction takes about 50 µs, so one core handles about 20,000 transactions per second per partition, and with one transaction at a time no concurrency control is needed.

Check

  1. Does the partition key match the transaction boundary? Each cross-partition transaction stalls the thread for a coordination round trip, so falls off a cliff as their share grows.
  2. Does the working set stay in memory? When disk I/O returns, transactions get long and the argument reverses.
  3. Can every transaction be a one-shot, non-interactive procedure?

Recommend

Adopt it only for workloads with a clean partition key, such as per-user or per-event state, and keep the rest in a general-purpose database.

Cost

Concurrency control has become a data-modelling decision you must defend permanently.

Read moreSingle-threaded per partitionThe boundary condition

8A mobile app calls your top-up API, and on a bad network the response is lost. How do you make the API safe, and what are the limits?Show the answer

Name the problem

The client cannot tell whether the top-up committed: the outcome is unknown.

Design

The client generates one per intent and sends it on every retry. The server inserts the key into a unique column inside the same transaction as the balance change. A duplicate key returns the stored result instead of applying the change again.

Limits

The key table needs a retention window, say 7 days. A retry after a key is evicted would be applied twice, so cap the client's retry period below the retention. Deduplication state also needs clean-up and monitoring.

Safety net

A reconciliation report against the payment provider, because the data pays quietly whenever the window assumption breaks.

Read moreThe in-doubt windowIdempotent consumers

9Six Spark jobs write to one Iceberg table, and 30% of their commits fail, costing hours of work. Diagnose and fix.Show the answer

Diagnose

The unit of conflict is a file, not a row. If each job touches about 100 of 10,000 files, any two jobs collide with probability near 1, and a taking about 50 ms makes each window of exposure long.

Check

Whether failed commits can be rebased: an append can simply retry its metadata step, but a MERGE or a compaction that rewrote files others read cannot.

Fix

One writer per table, or disjoint partitions per writer. Run compaction in a window without ingestion. Use append mode where possible. Retry with jittered backoff and a retry budget, and commit in fewer, larger batches: the ceiling is tens of commits per second.

Trade-off

Partitioning writers constrains the data model.

Read moreOptimistic commit on a metadata logWrite conflicts and retry semantics

10The CTO wants two-phase commit across the order database, the inventory database and an external payment API. Evaluate the idea.Show the answer

Capability check

The payment API has no prepare phase, so is not available across all three.

The numbers

For the two databases, locks are held for the local work plus two round trips and two log flushes, and with hot rows that duration enters the conflict rate squared. is the product of all participants, .

Recommend

A : reserve inventory, authorize the payment, then capture it, putting the irreversible step last, with compensating actions that are . Publish events through an , with idempotent consumers.

Failure modes

Intermediate states are visible (use a pending status), compensations can get stuck (route them to a human queue), and you need a reconciliation job.

Read moreSagasOrder the irreversible step lastWhat distributed transactions forbid

11Design the commit path for a database that must never lose an acknowledged commit, at 20,000 commits per second, on an SSD whose fsync takes 50 µs.Show the answer

The order

Write ahead and force at commit: the commit is the moment its record becomes durable. Release locks after the flush, and acknowledge last.

Estimate

Without batching, 1 ÷ 50 µs = 20,000 flushes per second, exactly the target, with no headroom. Use : its self-sizing batches let throughput rise with load.

An explicit option

For tables that can tolerate a tiny loss window, offer with a bounded window, as an explicit trade.

Clients

They still face the unknown-outcome window, so their retries need idempotency keys.

Read moreThe order of the steps is forcedGroup commit

12Events from an order database must reach 12 consumer teams. Choose between a transactional outbox and raw change data capture of the database's log.Show the answer

What they share

Both make the database commit the atomic event, and both deliver at least once, so consumers need idempotence either way.

Outbox

You design event shapes independently of your schema, at the price of a table with constant insert-and-delete churn (index bloat, pressure), eased by reading it through rather than polling.

Raw CDC

No extra table, but 12 teams become coupled to your physical schema: renaming a column breaks them.

Recommend

With 12 consumers and long-lived contracts, the outbox. Keep per-key order by partitioning events by order ID.

Read moreTransactional outbox

13A global database must provide external consistency. Compare commit wait with uncertainty restarts for a workload that is 95% reads.Show the answer

The same cost, billed differently

Both pay for the same clock uncertainty ε. (TrueTime-style) puts ε on every write as pure , and reads stay cheap. Hybrid logical clocks with an put it on reads, as restarts whose rate grows with clock skew.

Decide

With 95% reads, billing the 5% of writes is usually cheaper.

Also weigh

The cost of making ε small (GPS and atomic clocks, or a cloud bounded-time service) against the latency it saves. And correct the folklore: atomic clocks are not required, only a bound on uncertainty.

Read moreBounded clock uncertainty

14Your two-phase-commit coordinator crashed in production, and 3,000 rows are locked by prepared transactions. What now, and how do you stop this becoming an incident next time?Show the answer

Now

Recover the first: its log holds the decisions, and prepared participants cannot decide alone. If recovery is impossible, an operator makes , accepting a possible atomicity violation, and reconciles by hand.

Prevent

Replicate the coordinator with ; set timeouts and alerts on prepared transactions; keep lock-hold times short by repartitioning so most transactions are local; and design the reconciliation path before launch.

Know the default

Under , a transaction with no decision record is treated as aborted.

Read moreTwo-phase commitHeuristic decisionsPresumed abort

15A service writes to PostgreSQL and publishes to Kafka. After a restart, consumers see duplicate events. Is exactly-once possible here?Show the answer

Answer

Exactly-once delivery is not; is.

Design

Use an outbox so the database commit covers the intent to publish. Make consumers idempotent, by a natural key or a deduplication table whose retention exceeds how far back the broker can replay. For Kafka-to-Kafka stages, transactions commit the output and the input offset as one atomic event.

Watch

Deduplication entries expiring is the silent failure; monitor with reconciliation counts.

Read moreIdempotent consumersTransactional outbox