Scenarios
Longer problems, like those in senior system design interviews. A strong answer estimates with numbers first, names the mechanism, then recommends a fix and says what it costs.
1A wallet service reads a balance, checks it in Java, then writes the new balance. About 300 conflicting operations per second hit hot accounts, and you see rare lost updates. Diagnose and fix.Show the answer
Estimate
With 40 ms of application time and round trips, about 300 × 0.04 = 12 operations overlap each window, which is why it happens at all. A single SQL statement taking 40 µs would give 300 × 0.00004 = 0.012, a thousandfold reduction.
Mechanism
Read Committed allows the lost-update shape: one read–write edge plus one write–write edge.
Fix, cheapest first
SET balance = balance - x, guarded byCHECK (balance >= 0): it commutes, and the rule is a single-row constraint.- Otherwise, a version column and a retry loop.
- Otherwise,
SELECT … FOR UPDATE, accepting blocking and the risk of deadlock.
Failure modes
A retry loop needs an owner and a limit; locks must always be taken in the same order.
Recommend
The commutative update.
Read moreRead Committed: the home of lost updatesThe anomaly rate is Little's Law again
2Two operations each check that a room's total capacity would not be exceeded, then insert a booking. Occasionally the room is overbooked. The database runs at Repeatable Read. What is going on, and what are the options?Show the answer
Diagnose
Write skew: read a set (the bookings), check a rule over it, write into it. In PostgreSQL, Repeatable Read is snapshot isolation, whose conflict check sees nothing, because the two transactions inserted different rows.
Options
- Run those transactions at SERIALIZABLE and retry on serialization failure (error 40001). Cost: some false aborts and memory for tracking reads.
- Make the conflict concrete by updating a single
roomrow in both. Cost: false conflicts, and it silently stops working when a new code path forgets the update. - A unique constraint, if the rule can be expressed as one: enforced per row, and free.
Recommend
Option 1 for a few code paths, since this analysis does not compose across paths. On MySQL's InnoDB the locking behaviour differs, so test there.
Read moreWrite skew: one generative shapeRepeatable Read: three promises
3A new engineer says: 'We moved to a database that is serializable, so users will never see old data.' How do you respond?Show the answer
Correct the claim
Serializability limits which interleavings can happen, not how recent a read is. A read-only transaction may be ordered far in the past, and a replica may serve stale data. Recency is the other axis: single-object consistency. Strict serializability combines the two, and costs real-time coordination.
Practical advice
After a write, route the user's reads to the primary using a token, or use a system that offers external consistency, and measure replication lag.
Size it
With 667 operations per second and 2 s of lag, about 1,333 operations sit inside the stale window at any moment.
Read moreTwo hierarchies that must not be mergedExternal consistency
4Three services each use serializable transactions. A saga that spans all three shows inconsistent totals. Why?Show the answer
Diagnose
Serializability does not compose: independently serializable systems put together need not be serializable. Linearizable objects would compose, but transactions across services do not.
Options
Keep the invariant in a single database; add a coordination protocol; or redesign so each invariant lives inside one service and uses commutative operations.
Say plainly
That each service is transactional tells you nothing about the pipeline as a whole.
Read moreComposability decides architecture
5Choose isolation levels for a ledger with three kinds of work: transfers, a nightly report, and a rule that a joint account must keep a combined minimum. Justify each.Show the answer
Transfers
Restate them as commutative deltas on rows with CHECK (balance >= 0); then any level works.
Nightly report
It needs one consistent snapshot to avoid read skew, so snapshot isolation (or READ ONLY DEFERRABLE serializable). Read Committed would show totals that never existed.
Joint-account rule
A rule over a set of rows, so snapshot isolation is disqualified (write skew). Use SERIALIZABLE on that path and measure the abort rate, or a shared guard row.
Also
A vendor's name for a level must be tested, for example with a three-transaction test, not trusted.
Read moreWhat transactional isolation forbids
6After adding a read replica and a Redis cache, support tickets mention stale balances and missing history. Explain, and propose a plan.Show the answer
Explain
Each is a new place that can observe the data. The replica lags by 20 ms to 2 s; the cache serves values until their TTL expires. Expected anomalies ≈ rate of conflicting operations × window.
Plan
Read-your-writes, through a session token or by routing a user's reads to the primary for a short time after they write. Bound staleness and monitor lag. Keep serializable checks on the primary. Document which screens tolerate stale data.
Trade-off
More load on the primary.
Failure mode
Tokens lost when a request passes through a component that does not forward them.
7A wallet app expands from Frankfurt to Singapore, 80 ms apart one way. Product wants the balance screen to load in under 50 ms in Singapore, and transfers to stay correct. Design the read and write paths.Show the answer
Start from the arithmetic
Any linearizable or sequentially consistent path pays r + w ≥ 80 ms, so a 50 ms read in Singapore cannot be both linearizable and served from Frankfurt.
Options
- A single leader in Frankfurt for transfers (writes from Singapore take a 160 ms round trip: rare and acceptable), and the balance screen served from a Singapore replica with a stated bound, for example at most 2 s stale, with an alarm.
- Reads from the leader for the sender after a transfer, using a session token, so senders always see their own transfer.
- Lease-based local reads, if the product truly needs linearizable local reads, accepting an assumption about clocks that cannot be verified.
Estimate the window
At 667 conflicting operations per second and 2 s of lag, about 1,300 operations are inside the window.
Failure modes
The token is dropped at a load balancer (silently stale); the lag exceeds the bound (alarm, and fall back to the primary).
Recommend
Option 1 plus tokens, and never promise linearizable local reads in Singapore.
Read moreThe latency floorSession guaranteesFour ways to buy a fixed total
8A campaign gives a bonus to the first 5,000 customers across three regions. The limit must never be exceeded, and each region must keep accepting sign-ups during a network partition. Design it.Show the answer
Classify
"At most 5,000" is an upper bound that sign-ups push towards, so it is not invariant-confluent: coordination is required somewhere.
Design
Escrow. Split the 5,000 into regional quotas, for example 2,500, 1,500 and 1,000 by expected volume. Each region issues bonuses from its own quota with no cross-region call, and asks for more when it falls below a threshold.
Estimate
Coordination rate ≈ consumption ÷ slack held: a region using 5 per second and holding 1,000 refills about once every 200 seconds.
Cost
Stranded slack. During a partition, a region that runs out must refuse bonuses even though another region still has some. Reduce it with small refill blocks and a threshold-triggered refill.
Reject
"Count, check, then insert", at any isolation level: it is the write-skew shape again. Use a bounded counter, or a table of tokens decremented transactionally in each region.
Read moreInvariant confluenceThe one escape: partition the slack
9A distributed store is configured with N = 3, R = 2, W = 2, sloppy quorums and last-write-wins timestamps. A colleague says its reads are strongly consistent. Prepare a review.Show the answer
Why overlap is not recency
- R + W > N guarantees overlap only.
- Last-write-wins makes "latest" depend on clock skew.
- With no write-back after a read, monotonic reads can fail.
- A failed write can reappear later.
- Sloppy quorums let the W acknowledgements come from nodes outside the key's usual replicas, so the overlap argument fails exactly during failures.
Ask for evidence
Run a list-append or register test with a history checker.
Recommend by use
A ledger needs a linearizable store, or a leader with consensus. A feed needs a stated staleness bound and session tokens. Counters can use CRDTs.
Also
LOCAL_QUORUM quietly redefines N as one data centre, and a sequence of calls at mixed levels gets the weakest pairing.
Read moreTunable consistencySloppy quorumsWhat is actually verifiable
10Support reports that a user sees their new display name, then the old one again on the next refresh. Diagnose and fix.Show the answer
Diagnose
A failure of monotonic reads and read-your-writes, not a concurrency bug. Most likely a recent infrastructure change (a load balancer, a CDN, retries to another region) now sends requests to replicas at different positions in the log, and the client carries no token.
Test
Log the replica ID and its log position for every request.
Fix
A token: the commit position, carried in a header or cookie, so the router picks a replica that has applied it or falls back to the primary. Or sticky routing. Prefer the token: sticky routing turns a consistency guarantee into a load-balancing constraint that fights failover and autoscaling.
Make it owned
Name the owner of the token and add the guarantee to the SLO. Expected impact ≈ lag × request rate.
11Choose how to merge concurrent edits for three fields in a shared document store: the document title, the list of collaborators, and a view counter.Show the answer
Title
Last-write-wins is acceptable: any recent value will do, and its authority is the user who edited last. Warn that skewed clocks pick the winner, and a lost edit raises no error.
Collaborators
An observed-remove set (OR-Set), where an add wins over a concurrent remove. It costs tags and tombstones, so plan the collector and monitor the oldest uncollected tombstone: one offline replica blocks collection.
View counter
A grow-only counter (G-Counter), or a PN-Counter if decrements exist. Its metadata grows with the number of replicas, so avoid giving each client its own replica.
Convergence is not correctness
A rule such as "at most 10 collaborators" is an upper bound, not preserved by merging sets, so it needs escrow or coordination.
Read moreArbitrate: pick a winnerOR-SetG-CounterConvergence, not correctness
12Your CTO asks whether your two-region payments platform is 'CP or AP'. Answer, and say what you would measure.Show the answer
Reframe
The choice is made per operation, and only matters during a partition. Transfers and uniqueness checks, which are not invariant-confluent, must be CP: the minority side refuses them. Notes and counters can be AP, using CRDTs.
Measure what runs every day
The latency side of PACELC: p99 on a normal day for each operation, the distribution of replication lag, and how often the staleness bound is exceeded.
Also say
"CP" does not mean low availability: the system is unavailable only to the minority side, and only while partitioned. Identify out-of-band channels, such as SMS or phone calls, that need external consistency or a token-based wait.
Read moreCAP, stated preciselyPACELC
13You are asked to prove that your database really provides snapshot isolation, and to find any write skew. What do you do?Show the answer
Stance
Treat the vendor's level as a hypothesis to test.
Method
Build a workload whose version order can be recovered from reads, such as appending to lists. Record the history, infer the dependency graph with write–write, write–read and read–write edges, and search it for cycles. Do not search for a valid linearization: that is NP-complete.
Classify
A cycle with exactly one read–write (anti-dependency) edge is G-single, read skew, which snapshot isolation forbids. A cycle with two or more is G2, write skew, which snapshot isolation allows.
Report
Which cycle classes appear at which level, with the exact histories.
Caveat
A clean run does not prove absence; liveness and unobserved operations cannot be checked this way.
Read moreWhat is actually verifiableThe anomalies are three shapes
14A design review lists ten different consistency settings across an application's screens. How do you decide what each screen actually needs?Show the answer
An ordered procedure
- Try to make the operation commute.
- Check whether the rule fits inside one row.
- Check whether the rule is invariant-confluent.
- If it is not, try to partition the slack (escrow).
- If the application does read-modify-write itself, rule out Read Committed.
- If a rule over a set is checked and then written, rule out snapshot isolation.
- Ask whether any out-of-band channel is faster than replication.
- Only then choose the model, and write down the staleness bound.
The output
A table: screen, rule, the step that decided it, the choice, and who owns the token.
Ask of every cache and replica
How many observers, how far apart, and what does each believe?
Read moreWhat consistency forbidsApplying it screen by screen