Interview questions
Why recovery is a different problem
1Why can a caller not tell a slow service from a dead one?
Over a network, both send the same signal: silence. A timeout is a guess, not a measurement.
Read moreThe authority disappears
2What are the three classes of error, and which one is hard?
Permanent (it will never succeed), transient (it may succeed later), and ambiguous (the caller does not know whether the effect happened). Only the ambiguous class is hard.
Read moreThree error classes
3What are RPO, RTO and the duplication factor?
The recovery point objective (RPO) is how much data you may lose; the recovery time objective (RTO) is how long you may be down; and the duplication factor is how many times recovery may repeat an effect.
Read moreThree quantities
4Why is availability dominated by repair time?
A = MTBF / (MTBF + MTTR), mean time between failures over that plus mean time to repair. Dividing repair time by 10 gives the same gain as making failures 10 times rarer, and it is usually far easier.
Recovery mechanisms
5How often should a long-running job checkpoint?
At I* = √(2 · C · MTBF), where C is the time a checkpoint costs and MTBF the mean time between failures. With C = 30 s and one failure a day, about every 38 minutes.
Read moreThe optimum
6Why does exponential backoff double the wait each time?
It is a search for an unknown recovery time: doubling overshoots by at most a constant factor and needs few probes to find it. Reducing load is a secondary benefit.
7Why must retries include random jitter?
The failure itself synchronizes the clients: they all fail at once, and without jitter they all retry at once. Randomness spreads them out without any coordination, and full jitter also shortens the total time to finish.
Read moreWhy jitter is not optional
8What is a retry budget, and why is it needed?
A cap on retries as a fraction of live traffic, for example 10%. Without it, retries at each hop multiply down a call tree: three tries at each of three levels is 3 × 3 × 3 = 27 requests at the bottom.
Read moreThe retry budget
9What must be true for replaying a log to reproduce the same result?
The processing must be deterministic: no reading the wall clock, no random numbers, no lookups into data that has changed since.
Read moreDeterminism is the precondition
10After an outage, when does a pipeline catch up, and when does it never catch up?
Recovery time = backlog ÷ (throughput − arrival rate). If the pipeline's throughput does not exceed the arrival rate, recovery never ends.
Read moreThe catch-up condition
11Why is recomputing lost data from its lineage expensive across a shuffle?
Each output partition of a shuffle depends on every input partition, so losing one forces re-running the whole upstream stage. That is why systems keep shuffle files around as a de facto checkpoint.
Read moreThe cost is the fan-in
12Why does failover need fencing?
The old primary may be slow, not dead, and can never be reliably told it was replaced. Only the shared resource can reject its writes, using an epoch number that increases with each new primary.
Making a repeat harmless
13Why can a sender never prevent a duplicate?
After a timeout it cannot know whether the effect happened: every acknowledgement would itself need acknowledging (the Two Generals problem). A duplicate can only be absorbed where state is kept, at the receiver.
14What does idempotent mean, and what part of it is easy to miss?
Applying an operation any number of times leaves the same observable state as applying it once. Easy to miss: the response must also be repeatable, or the caller still sees a different outcome on the retry.
Read moreThree sources of idempotence
15Idempotence comes from three sources. Name them and their costs.
Natural: the operation's algebra absorbs repeats (setting a value), and it is free. Positional: each write goes to a deterministic address, so a repeat overwrites itself, which is cheap. Imposed: a record of what was already done, which is expensive state.
Read moreNatural idempotencePositional idempotenceImposed idempotence
16Why is balance = 110 safe to retry while balance += 10 is not?
Absolute operations are idempotent; relative ones are not. But converting a relative operation into an absolute one needs a read first, which brings back lost updates and the need for compare-and-set.
Read moreNatural idempotence
17Who must generate an idempotency key, and when?
The client, before the first attempt, reusing it on every retry. The key names the intent, not the attempt.
Read moreImposed idempotence
18Why must the receiver store the outcome of a request, not just suppress the duplicate write?
Otherwise a retry gets an error, and the caller cannot tell "it was already done for me" from "someone else did something".
Read moreImposed idempotence
19When is a deduplication window correct, and when does it fail?
It is correct only if the window is longer than the longest delay a duplicate can have. It fails when the system is slow, which is exactly when retries happen.
20Why is a Bloom filter dangerous for deduplication?
A false positive says "seen before" about a new record, which is then silently dropped. Use a Bloom filter only as a pre-filter in front of an exact check.
21Is exactly-once delivery possible?
No, and asking for it is a category error. Delivery is at-least-once or at-most-once. An exactly-once effect is achievable, but only within the set of effects a system can commit atomically together.
Read moreThe exactly-once category error
22What does a transactional sink require, and what timing condition must hold?
It needs a write that is durable, invisible to readers, and can be committed later. And the sink's transaction timeout must exceed the checkpoint interval plus detection, restart and draining the backlog; otherwise the sink aborts the transaction before the engine can commit it.
Read moreTransactional sinksThe prepared transaction has a timeout
23A consumer can commit its offset in only two orders relative to processing. What are they, and what is the escape?
Before processing, which is at-most-once, or after, which is at-least-once. The escape is to store the offset in the sink itself, in the same transaction as the effect.
Read moreConsumer offsets and commit ordering
24With out-of-order processing, why must a consumer commit the low-water-mark offset?
When records finish out of order, committing the highest completed offset skips earlier records that are still unfinished, and they are lost on a crash. Commit only up to the point below which everything is done.
Read moreOut-of-order completion
25What does deduplicating by sequence number give that a set of seen keys does not?
State proportional to the number of producers rather than messages, and loss detection: a gap in the sequence shows something is missing. It needs ordering per producer, and a producer identity that survives restarts.
Read moreDeduplication on sequence numbersSequence numbers also detect loss
26Why is a dead-letter queue weak on its own?
It silently makes downstream data incomplete. Treat the queue's depth as a data-quality signal, with an owner.
27What is wrong with Kafka's default auto-commit of consumer offsets?
It commits the fetched position on a timer, without knowing whether processing finished. If the consumer crashes, records that were fetched but not yet processed are lost, with no error. It is at-most-once that looks like at-least-once.
28What is wrong with deduplicating by first checking whether a key was seen, then acting?
Two consumers can both check, both find nothing, and both act. The check and the record of having seen the key must be one atomic operation: INSERT … ON CONFLICT, Redis SETNX, or a unique constraint.
Read moreThe concurrency problem
29Why does Kafka's exactly-once mode not cover a write to Postgres or a call to an HTTP API?
Atomicity needs one system that decides whether all the effects happened. Kafka's transactions decide only about writes into Kafka; Postgres and the API each have their own commit point, which Kafka cannot include.
30How do you handle an external effect that cannot be deduplicated, such as sending an email?
Decide for each effect, with its business owner, which is worse: a possible duplicate (at-least-once) or a possible absence (at-most-once). Then shrink the window of doubt: send through an intermediary that can deduplicate, perform effects in order of how reversible they are, and record the intent before acting.
Redundancy and availability
31What is a witness replica, and what does it cost?
A member that votes but stores no data: about 2× storage instead of 3×. But losing one data replica leaves no redundancy at all while it is rebuilt, and the witness must sit in a third failure domain.
32Why is a write accepted through hinted handoff not a real quorum write?
It was acknowledged by nodes that do not own the key, so R + W > N no longer guarantees that a later read overlaps it.