Interview questions

• Rate: they run at different speeds. Decoupled by a buffer, which costs memory or disk, latency and staleness.
• Availability: one can be down while the other runs. Decoupled by persistence, which costs durability work, acknowledgement latency and clean-up.
• Identity and lifetime: they need not know each other or exist at the same time. Decoupled by an addressing indirection such as a topic, which costs coordination and more to operate.

Read moreWhat data movement actually is

Where the progress state lives. A queue's broker tracks every message and deletes it once a consumer acknowledges it. A log's broker tracks nothing per message: each consumer keeps its own position (an offset), and records are deleted only when the retention period runs out. Everything else, such as replay and many independent readers, follows from that.

Read moreQueue versus log

Emails are independent tasks of varying cost, with no value in keeping their history, so you want per-message parallelism and retry: a queue. Orders are facts that many consumers need and that you will need to reprocess one day, so you want order per key and the ability to replay: a log.

Read moreWhen to pick which

A log is consumed in order, so one record that cannot be processed stalls everything behind it in its partition. The options:

• block and retry, which suits transient failures only;
• skip and move on, recording the rejected record somewhere;
• move it to a retry topic, which breaks per-key order;
• pause that partition and alert someone.

Read moreThe head-of-line problem

Durability protects a message until it is consumed; it survives a broker crash. But once acknowledged, it is deleted, so it cannot be read again. Replay needs retention, keeping records after they are consumed, which is what a log provides.

Read moreEight transports on three switches

You cannot. A lost request, a request processed whose reply was lost, and a server that crashed halfway all look identical to the caller. The only fix is to make the receiver idempotent and have the caller send an idempotency key, so a retry is safe whichever case occurred.

Read moreThe epistemic limitRequest/response and RPC

Push gives low latency and simple consumers, but the sender must track every consumer and needs flow control to avoid overwhelming them. Pull lets each consumer take batches at its own pace and gives backpressure for free, at the cost of a latency floor of one poll interval. Pull is the safer default when traffic is unknown; long polling gives pull with nearly push latency.

Read morePush versus pull

Only when the operations do not commute: deltas such as "add 5", or inserts and deletes of the same row interleaved. Full-state updates that carry a version do not need order, because the consumer can keep the highest version. Converting deltas to full state often removes the ordering requirement altogether.

Read moreOrdering, precisely

It keeps only the latest record for each key, so the log's size grows with the number of keys, not with time, and the log effectively becomes a table. Consumers that need the history of changes, not just the latest state, become silently wrong.

Read moreLog compactionThe semantic cost

Four, and only four:

• absorb it: buffer or batch;
• slow the source: backpressure, rate limiting, throttling or pacing;
• drop some of it: load shedding or admission control;
• cut the path: a circuit breaker or a bulkhead.

A buffer only buys time: if arrival stays above service, it fills.

Read moreArrival exceeds service: four answers

It accepts work that will never be processed in time, so records are stale by the time they are handled, and it ends in an out-of-memory crash anyway, losing more work than a bound would have. Every buffer needs a size limit and a declared policy for when it is full.

Read moreBuffering

A large buffer that stays full. It adds the delay of a full buffer to every record and no throughput at all, because throughput is set by the consumer's rate. It also hides overload from monitoring, since nothing fails. Size buffers to absorb bursts, not sustained overload.

Read moreBufferbloat

Both, whichever comes first. Size alone gives unbounded latency when traffic is low, since the batch may never fill. Time alone gives unbounded batch size under a burst. Set the time limit from your latency target, and the size limit from the unit the downstream system handles comfortably.

Read moreSize-based, time-based, both

A token bucket limits the average rate but allows bursts up to its capacity: tokens accumulate while traffic is quiet and can be spent at once. A leaky bucket drains at a fixed rate, so its output is smooth whatever the input looks like.

Read moreRate limiting

When the source cannot safely wait: a person, a partner's system, a fleet of devices, or a database that is harmed by being paused. Backpressure keeps travelling upstream until something gives way. In those cases use load shedding, admission control, or a durable buffer that absorbs the excess.

Read morePropagation

Goodput is useful completed work, such as responses delivered before the client gave up. Past saturation, throughput can stay high while goodput collapses, because work that will be thrown away crowds out work that would have been used. Shedding load keeps goodput flat.

Read moreLoad shedding

Admission control refuses work before any resources are spent on it; shedding drops work that was already accepted. Admission is cheaper, and shedding corrects for admission's wrong estimates, so use both.

Read moreAdmission control

Rejecting a request early costs almost nothing, so goodput stays at full capacity. A request that times out has already cost its full service, so timeouts can drive goodput to zero.

Read moreLoad shedding

Balance (even load), locality (related records together), order (sequence kept within some scope) and stability (assignments survive destinations joining or leaving). You cannot maximize all four; balance and locality pull directly against each other.

Read moreWhich of N destinations gets this record?

Records are facts about entities, and most processing is per entity. Partitioning by key puts all of an entity's records in one partition, which keeps them in order, keeps per-entity state local to one consumer, and lets two data sets partitioned the same way be joined without moving data.

Read moreKey-based partitioning

Look at traffic per key. If one key dominates, it is key skew: salt the key, aggregate in two phases, give the key its own path, or choose a different key. If no single key dominates but a partition is still overloaded, it is partition skew: add partitions or improve how keys are assigned to them.

Read moreSkew

With one position per node on the hash ring, the arcs are uneven, and a node that leaves hands all of its load to one neighbour, which can fail in turn. With 100–256 positions per node, the load evens out (the spread shrinks as 1 / √V for V positions), a departing node's load spreads over many survivors, and giving nodes different weights is easy.

Read moreVirtual nodes

For tens to a few hundred nodes. Rendezvous hashing scores every node for each key and picks the highest: no ring state, no virtual nodes, minimal movement on membership change, a natural ordered list of replicas, and about ten lines of code. Its lookup costs O(N), so at very large N use a hierarchical version or a ring.

Read moreRendezvous hashingWhen to use which

• The partition count changed, so the key moved.
• A producer retried with several requests in flight and no idempotence, so a retried record landed after a later one.
• The consumer processes with a thread pool or asynchronous side effects.
• A record went to a dead-letter queue and was re-injected later.
• Several producers write the same key.
• The key itself was changed.

Read moreWhere it breaks in practice

It turns an availability problem (the pipeline is stuck) into a data-completeness problem (some records are missing downstream), and an ordering guarantee into a best-effort one. For a log, it locally turns the log back into a queue.

Read moreDead-letter queues

At least the target throughput divided by one consumer's throughput, multiplied by 2–3 for headroom; at least the largest number of consumers you will ever want; and preferably a number with many divisors, so it splits evenly across different consumer counts. Err slightly high: changing it later remaps keys and is hard.

Read moreChoosing N

TCP delivers bytes in order on one stream, so one lost packet stalls every HTTP/2 stream sharing the connection. QUIC makes the transport aware of separate streams.

Read moreThe same shape, one layer down

A key cannot be divided and its data must live together, so a key bigger than one node's capacity always lands on one node.

Read moreHot keys and data-dependent skew

Only when the aggregation is associative and commutative, such as sums and counts, so the partial results from each salted piece can be combined afterwards.

Read moreThe escapes and what each gives up

Trusting the declared length. A frame that claims to be 4 GB makes the receiver allocate 4 GB and crash, or wait forever for bytes that never come. Always enforce a maximum frame size before allocating anything.

Read moreLength-prefixed frames

Compare times, not ratios: compress + send the compressed bytes + decompress against sending the uncompressed bytes. Compress when the link is the bottleneck, as across regions or with metered egress. On fast links inside a data centre the CPU is usually the bottleneck, so use a fast codec or none.

Read moreA rate comparison, not a ratio comparison

A shorter linger time means smaller batches, and smaller batches compress worse, because compression finds redundancy across records. Batch size is also a compression setting.

Read moreThe interactions

NAT gateways, firewalls and load balancers silently drop connections that have been idle too long, and the pool does not know. Keep the pool's maximum idle time below the shortest idle timeout on the path, validate connections when they are borrowed, or retry once, for idempotent calls only.

Read moreConnection pooling

Nagle's algorithm holds back a small packet until the previous one is acknowledged; the receiver delays its acknowledgement hoping to piggyback it on data. Each waits for the other until a timer fires, about 40 ms. For request/response traffic, set TCP_NODELAY and do your batching in the application.

Read moreNagle-style coalescing

Delivery is how many times the transport hands a record to the consumer. Processing is how many times its effect happens in the world. The practical target is at-least-once delivery with effectively-once processing.

Read moreDelivery versus processing

Non-determinism: reading the current time, random numbers, the iteration order of hash maps, or lookups into data that has since changed. A pipeline can be replayed only if its output is a pure function of the inputs it retains.

Read moreThe preconditions