Interview questions
No shared present
1Why can a failure detector never be perfectly accurate in an asynchronous network?
A slow node and a dead node look identical from outside: neither sends anything. Any timeout trades how fast you detect real failures against how often you wrongly suspect live nodes.
Read moreThe reframeFailure detection
2What is the difference between safety and liveness in a distributed protocol?
Safety means nothing bad ever happens, such as two leaders acting at once. Liveness means something good eventually happens, such as a leader being elected. Safety comes from quorums overlapping and uses no clocks; liveness comes from timeouts.
Read moreSafety and liveness
3Why is getting nodes to agree expensive, while spreading information is cheap?
Agreement needs round trips to a quorum, and stops when a quorum cannot be reached. Dissemination can use gossip or caches, may be stale, and needs no decision at all.
4What does the FLP impossibility result say, and how do real systems get around it?
In a fully asynchronous system where even one node may crash, no deterministic algorithm can guarantee that consensus is always reached. Real systems escape by assuming the network is eventually well-behaved (timeouts and leader election), or by using randomization.
Deciding without observing
5Why do majority quorums guarantee safety?
Any two majorities of the same N nodes share at least one node (pigeonhole), so two conflicting decisions can never both be accepted: the shared node would have to accept both.
Read moreConsensus
6Why does a consensus group need 2f + 1 nodes to survive f crashes?
A decision needs a majority, f + 1 responses, and f nodes may be silent. So N − f ≥ f + 1, which gives N ≥ 2f + 1.
Read moreWhy 2f+1
7Why does adding nodes not make consensus faster?
Every decision still passes through one leader and a majority of nodes. More nodes make the majority larger and each round slower; they add fault tolerance, not throughput.
Read moreWhy it does not scale out
8What is an epoch (or term) in leader election, and why does it matter?
A number that increases with every change of leader. Followers and storage reject anything stamped with an older epoch, which stops a former leader that does not yet know it was replaced.
Read moreLeader election and epochs
9Why is a lease with a time-to-live not a lock?
The holder can pause, for a garbage collection or a slow disk, beyond the expiry and still believe it holds the lock, while someone else now does. The protected resource itself must check a fencing token: a number that increases with each grant, so writes carrying an older one are rejected.
Read moreLocks, leases and fences
10Why can a two-site deployment not fail over automatically and safely?
Neither site can tell a dead peer from a cut link, so either both promote themselves (split brain) or neither does. A small witness in a third failure domain breaks the tie.
Read moreSplit brain
11Why should cluster membership be changed one node at a time?
Going from 3 nodes to 5 in one step can create two majorities that do not overlap: 2 of the old 3 and 3 of the new 5. Changing one member at a time keeps every old and new majority overlapping. Joint consensus is the general alternative.
Distributing data
12Why does adding partitions not fix a hot key?
A partition function maps each key to exactly one partition. Doubling the partitions moves other keys around, but the hot key still lands in one place. Salt it or replicate it instead.
Read morePartitioning and the partition count
Communication
13Why does a remote call have three possible outcomes, and what do you do about it?
Success, failure, and unknown (no reply). The unknown case cannot be resolved, so make it harmless: request IDs generated by the client, with a deduplication window on the server, make retries idempotent.
Read moreRPC: idempotency and deadlines
14Why should a request's deadline be absolute and passed along to every service it calls?
Relative timeouts restart at each hop, so they can add up to many timeouts. An absolute deadline gives each hop only the time remaining. And by Little's Law (L = λW), capping the time W a request can spend caps the number in flight, so it doubles as free admission control.
Read moreRPC: idempotency and deadlines
15What is in-band coordination, and what does it require?
Facts that travel on the request itself (a deadline, a trace ID, the tenant, an epoch, an idempotency key), so they need no extra round trip. They must be interpretable by the receiver without any shared context.
Read moreIn-band coordination
16When does hedging (sending a duplicate request after a delay) help, and how can you tell it is not helping?
Hedging helps only when slowness belongs to the attempt (one slow server), not to the request itself (an expensive query). If more requests are hedged than the configured percentile implies, for example more than 5% when hedging at p95, the slowness is in the requests.
Read moreHedging
17Why does a request that fans out to 100 shards have a bad tail?
Its latency is the maximum of 100 shard latencies: if each shard is slow 1% of the time, 1 − 0.99¹⁰⁰ ≈ 63% of requests hit at least one slow shard. Availability multiplies too: 0.999¹⁰⁰ ≈ 90%.
Read moreScatter-gather
18What are the two rules for separating a control plane from a data plane?
The control plane must not sit synchronously on the data path: the data plane keeps working on its last known configuration if the control plane is down (fail static). And control-plane load must grow with the rate of changes, not with the rate of requests.
Read moreControl plane and data plane
19Every way of resolving a coordination problem is paid for in one of three currencies. What are they?
Waiting, wasted work, and weakened semantics. A design that seems to pay none of them has hidden one, usually weakened semantics.
Read moreThe consolidated escape table