Summary and key takeaways

From one machine to two: what changed

• Three independences (rate, availability, identity/lifetime) each cost something to decouple: buffering, persistence, an addressing indirection.
• RPC decouples none of the three; a durable log decouples all three. Place any transport on these three switches.
• The impossibility behind all of it: you cannot tell "not done" from "done, but the ack was lost".
• The four groups (transport, routing, flow control, network mechanisms) are a stack. A pipeline chooses one cell in each at the same time. Kafka is a composite, not a primitive.

Transport patterns: transport patterns

• Queue versus log is about where progress state lives: the queue's broker tracks each message, the log's consumers keep offsets.
• The log's real tax is head-of-line blocking; the fix is to pause the partition and side-line the bad record.
• Pull gives structural backpressure and natural batching; push gives low latency and a flooding risk.
• Process-then-ack gives at-least-once; fix duplicates with an idempotent sink. Use a task queue for jobs and a log for facts.

Flow control: flow control

• When arrival exceeds service rate you have four answers: absorb, slow the source, drop, sever. Every buffer needs a full-policy.
• Backpressure, rate limiting (token bucket), load shedding, admission control, circuit breakers and bulkheads each answer overload differently; composed carelessly, retries and timeouts amplify an outage.

Routing and distribution: routing and distribution

• Partitioning buys parallelism and per-key order, and costs global order. The key choice is the real design decision.
• Skew (key skew versus partition skew) cannot be fixed by flow control or more consumers; use salting, splitting, better keys.
• Consistent and rendezvous hashing keep assignment stable when nodes change. A DLQ must classify failures and never drop silently. Choose N for the future peak, because changing it later is a migration.

Network-level mechanisms: network-level mechanisms

• Framing, compression, pooling, keep-alive, zero-copy, Nagle and TLS are mostly constant-factor choices you look up, but some change behavior: compression is a rate comparison, idle pooled connections die silently without heartbeats, and layered optimizations (Nagle plus delayed ACK) compose badly.
• Benefit from the rest by not defeating them.

Delivery across a pipeline: delivery across a pipeline

• Delivery is a transport property; processing is a property of your consumer and sink. Aim for at-least-once delivery and effectively-once processing.
• Loss, duplicates and reordering each have a few causes at a crossing; duplicates and reordering can be repaired downstream, loss cannot. Making a repeat harmless is fault tolerance's subject.
• Replayability is architecture: retention, determinism, idempotent sink, resettable state. Chains have the semantics of their weakest hop; only endpoint guarantees are real.
• Verify with reconciliation counts, a duplicate-rate metric and deliberate crash tests.

Laws to remember

1. Every guarantee has a scope. Ask where it ends.
2. Costs are conserved and relocated, never removed. Decoupling is relocated coupling: async turns a call dependency into a shared retention window and dedup contract.
3. Latency versus throughput is the master trade-off; batching trades one for the other.
4. Ordering versus parallelism and push versus pull are hard to reverse.
5. Durability versus ack latency: "sent", "buffered" and "fsynced on N replicas" are different promises.
6. Variance costs more than volume; local optimizations compose into global pathology (retry amplification, metastable states).
7. Guarantees compose to the weakest link; atomicity is bounded by the commit substrate; correctness under retry belongs to the effect, not the transport; determinism is an asset that decays silently.

Three questions that prevent the worst incidents

1. What is the full-policy of every buffer?
2. Where do you deduplicate: name the key and name the store?
3. What happens to a record that matches no route, or can never be processed?

How movement fails. Failures are quiet: duplicates look like data, a stalled partition looks like a metric with nothing to report. So monitor for absence and asymmetry: counts that do not reconcile, a lag that stopped changing, an age that only grows. Movement decisions (partition key, retention, ordering) are made early and bind every later consumer, so spend disproportionate design time here.

The competence. Systems that feel solid are not the ones that lose nothing; they are the ones where every loss was chosen, bounded and visible. Be able to look at a running pipeline and say exactly what it is currently willing to lose, and whether anyone decided that.