Scenarios

Orders are facts, so use a log with 7–30 days of retention, partitioned by order_id so each order's events stay in order.

1,000 records/s × 1 KB = 1 MB/s, or 86.4 GB a day: about 605 GB for 7 days in one shared log. Five queues, one copy per subscriber, would write 5 MB/s and store about 3 TB for the same backlog.

The shared log carries the facts. Keep a separate task queue for per-message work, such as emails and image jobs.

All subscribers share one retention period, so document it as a contract: shortening retention shortens every consumer's tolerable outage.

A consumer that falls further behind than the retention period loses data, so alert on consumer lag as a fraction of retention.

Suspect head-of-line blocking by a poison record: the consumer is retrying one record forever. Check the committed offset of that partition (it does not move) and the consumer's error logs. Each hour of stall adds 360,000 records to the backlog.

Retry a bounded number of times, then write the record and its error to a reject topic, move on, and alert. For transient causes, pause only that partition.

Per-key order is broken for the set-aside record, so downstream consumers must tolerate its late arrival, for example by applying versioned upserts.

Process-then-acknowledge is at-least-once: a crash or a rebalance between the two, which deployments cause, redelivers the message.

Fix the effect, not the delivery. Make stock updates full-state upserts keyed by (order_id, line), and record sent emails under a unique (order_id, kind) key. For each effect, name the key and the store that holds it.

Sending an email and inserting its record are still two steps, so a rare duplicate remains unless the email provider accepts idempotency keys.

No added latency, but anything not yet replicated is lost on failure: a loss window equal to the replica's lag.

Adds about one round trip (0.5–2 ms across availability zones, 0.2–0.5 ms within one) and usually loses nothing. Check whether the replica writes to disk before acknowledging, or only receives.

Adds the round trip to the slowest replica needed, and stalls if fewer than a quorum are available.

A quorum within one region (no loss window) and asynchronous replication to a second region, because a 70–250 ms intercontinental round trip on every commit is not acceptable for interactive writes. Say plainly that stalling below quorum is the price in availability.

With a 1 ms round trip, one connection handles at most 1,000 records/s. 50 million a day averages about 580 records/s, close to that limit, so any slowdown makes the service fall behind. Each call also ties the service's availability to the API's.

Send 100 records per call, raising the ceiling to about 100,000 records/s. Accept that the first record waits for its batch and that a batch fails as a unit, and add an idempotency key per batch.

If the result is not needed immediately, publish the records to a log and let a consumer call the API at its own pace, with its own batch size.

End-to-end exactly-once across independent systems is impossible. Promise effectively-once, designed separately for each destination.

• Search index: an idempotent upsert by order_id plus a version.
• Billing: an idempotency key chosen by the producer before the first attempt, with a deduplication window longer than the longest replay gap.
• Email: at-least-once with a unique (order_id, kind) record, accepting a rare duplicate.

Kafka-to-Kafka steps can use transactions for true exactly-once, but the guarantee ends at the first external call.

The backlog at the end of the peak is (2,000 − 800) × 7,200 s = 8.64 million events. Afterwards it drains at 800 − 300 = 500 per second, so data stays stale for 17,280 s, about 4.8 hours, if the consumer stays at 800 per second.

Let the durable broker be the place that absorbs the burst: backpressure stops there, with the consumer pulling. Scale out consumers and batch adaptively to raise throughput above 2,000 per second, and decide which data may be late. Where newer data supersedes older data, drop the oldest per key.

Measure lag in seconds, not records.

Unbounded in-process buffers (out-of-memory crashes, silent loss), and consumers that acknowledge before the data is durable.

The partner is an upstream you cannot apply backpressure to; your retries are adding to its overload, and its slowness is holding your checkout.

• A client-side token bucket at the partner's documented limit, honouring Retry-After.
• Full-jitter backoff and a retry budget of about 10% of calls.
• Timeouts set from the partner's p99.9.
• A circuit breaker on the error rate, with a jittered half-open state, and a fallback: accept the address unvalidated and validate it asynchronously.

The fallback lets some bad addresses through. And retries must count as evidence of failure, or the breaker never trips.

Backpressure travelled upstream: the consumer stalled, the replication slot stopped advancing, and the database kept all of its write-ahead log.

End the backpressure at a durable buffer, such as a log, that reads the slot continuously. Monitor slot lag and WAL size with alerts, and set a maximum retained WAL, accepting that the slot can be invalidated and the pipeline re-snapshotted.

Backpressure is safe only towards an upstream that can be controlled and can afford to wait.

Most likely bufferbloat plus a retry storm: queues are long, clients time out and retry, and goodput collapses while errors stay low.

Add admission control that limits concurrency (derive the limit from Little's Law, or adapt it), shed by deadline and priority at the edge with 429 plus Retry-After, shrink the buffers, and cap retries with a budget.

Alert on queueing delay and goodput, not on error rate.

Token buckets, with the burst capacity chosen on purpose. Each server keeps local buckets that periodically lease tokens from a central store, so there is no round trip per request and the system degrades gracefully. Return 429 with Retry-After.

What happens when the central store is down: failing open with a local fallback limit is usually safer than failing closed.

The limit is approximate, and a tenant whose traffic concentrates on one server needs lease sizes that follow demand.

10 ms collects about 50 events (25 KB) per batch; 500 ms collects about 2,500 (1.25 MB). The fixed cost per request, a round trip plus broker work, is shared over 50 records or over 2,500.

The gain usually flattens long before 500 ms. If the latency target is under 100 ms, send on size or age, whichever comes first, with an age of 10–20 ms.

A larger batch means a bigger retry when it fails, and more memory held while it fills.

A partition is read by one consumer at a time, so extra consumers cannot help it. Check the key histogram and per-partition lag: one key dominating means key skew, not partition skew.

At 2,000 events/s the hot seller sends 700/s to one partition. If one consumer handles 400/s, the backlog grows by 300/s, 1.08 million an hour.

1. Ask what order the business actually needs. If it is per order, re-key by order_id: high cardinality, even spread, and it matches the real ordering scope.
2. For per-seller aggregates, aggregate in two phases: pre-aggregate across partitions, then combine the small partial results by the true key.
3. If per-seller order is really required, give the hot seller its own topic and consumer, or salt its key if the logic is associative and commutative.

Re-keying is a migration: a new topic, writes to both, then a cut-over. Salting gives up per-key order and needs a second stage. Either way, add an alert on the maximum partition lag.

Tenant alone gives skew. Use a composite key, (tenant_id, entity_id), so large tenants spread across partitions while each entity's events stay in order.

Tier them: a shared topic for the long tail, dedicated topics for the large or noisy tenants. A shared topic means shared fate, since one tenant's burst or poison record affects all; but 10,000 topics is an operational problem of its own. Route by headers or topic name, not by parsing payloads.

Choose the partition count from throughput and consumer limits, round to a number with many divisors, and over-partition slightly.

A growing tenant must be promoted from the shared topic to a dedicated one. Plan the re-routing window, during which its events may be reordered.

Changing N in hash % N moves about (N − 1) / N of the keys: going from 20 to 21 nodes moves about 95%, so almost everything misses at once.

Use rendezvous hashing (with 20 nodes its O(N) lookup is trivial; no virtual nodes; the ranking gives replica and failover order for free), or a ring with 100–256 virtual nodes. Now one membership change moves about 1/N of the keys, about 5%, and a dead node's load spreads across all survivors instead of toppling one neighbour.

If a few keys are very hot, cap each node at (1 + ε) times the average load (bounded loads).

Coalesce requests and rate-limit during warm-up. The cost is slightly more complex client logic; test it with a drill that kills a node.

• Was the key the same (order_id) for both events, and unchanged?
• Were both written by one producer session, or by two services?
• Is producer idempotence on, so a retried Charged cannot land after Refunded?
• Does the consumer process with a thread pool or asynchronous writes?
• Was the partition count increased, remapping keys?
• Did a failure send Charged to a dead-letter queue, from which it was replayed later?

One writer per key, an idempotent producer, single-threaded processing per partition, and never changing the partition count in place. If a record fails and order matters, pause the partition instead of dead-lettering it.

Is order needed at all? Carry a state version and let the consumer apply the latest version, converting deltas to full state.

• Transient (503, timeouts): retry with exponential backoff and jitter; never dead-letter.
• Permanent, bad data (a malformed record): dead-letter at once, with the payload, its topic, partition and offset, the error, the attempt count and the consumer version.
• Systemic (the whole API is down): open a circuit breaker, pause consumption, alert, and let the broker hold the backlog.

If needed, use retry topics (retry-5s → retry-1m → retry-30m → DLQ), accepting that retried records arrive out of order for their key.

Stop if more than a small share of recent records fail: at 5,000/s, a few percent is over 100 dead-lettered records per second, which is an outage, not poison. Monitor the queue's depth and age, and give it an owner.

Consumers must be idempotent, since replays reprocess partly applied work; test replay. Pausing a partition delays everything behind it, but preserves order and keeps the stream from draining into the dead-letter queue.

Not for the same key. If one order's events land in different priority queues, their order is lost.

• Assign priority per order: all of an order's events go to the same queue, so each order stays in order while tiers differ between orders.
• Use separate queues read with weights to prevent starvation: 10 : 3 : 1 gives gold about 71%, silver 21% and bronze 7% of capacity.
• Apply priority mainly at admission: reserve capacity for gold and shed bronze under overload.

Priority matters only when the system is saturated; if it is needed all the time, the system is under-provisioned. Cost: P priorities × K partitions is P·K queues to monitor, plus a policy for consuming across them.

The raw stream is 1 MB/s, about 86 GB a day; at a per-GB egress price that is a large line item.

Compress batches, not records: with 1,000-event batches expect several times the ratio (for example 6× instead of 1.2×). Raise the producer's linger time until batches are large, within the latency budget.

Zstd at level 1–3 on the cross-region link, which is bandwidth-bound; LZ4 or nothing inside the data centre, which is CPU-bound. Make sure the broker stores batches as sent, without recompressing them, and consider a trained Zstd dictionary if records stay small.

A profile full of TLS handshakes means connections are not being pooled.

Larger batches add latency, up to the linger time, and more data is lost if a producer crashes. Fewer partitions improve the ratio but cap parallelism.

During the quiet night, a NAT, firewall or load balancer silently dropped idle connections, and the pool hands out dead ones. The first requests fail or wait for a timeout, then reconnect, paying a handshake of 1–3 round trips.

The pool's idle time against the shortest idle timeout on the path; reset or timeout errors on the first requests of the day; handshake counts.

Keep the pool's maximum idle time below the shortest timeout, send application-level keep-alive pings, validate connections on borrow (one extra round trip) or retry once for idempotent calls, and set a maximum connection lifetime so connections are recycled.

Retrying non-idempotent calls on stale connections.

A heartbeat thread (for example every 3 s, with a 30 s session timeout) shows the consumer is alive; a separate limit on the time between polls, sized to the longest legitimate batch, shows it is making progress.

Declaring a live consumer dead triggers a rebalance that pauses all 200 consumers, and rebalances can trigger more rebalances: a failure that sustains itself.

A longer session timeout, static membership if available, and cooperative rebalancing so only the affected partitions move. For a home-grown detector, use phi-accrual detection, which sets a suspicion level from the observed distribution of heartbeat intervals, instead of a fixed timeout.

A consumer that really is dead is detected more slowly, and its partitions stall for up to the session timeout.

To add the header, the proxy probably deserializes every message, and may decompress and recompress it. That destroys the path's value as a plain conduit: fetches lose zero-copy, and CPU goes on a full compression cycle per message.

Put the header where the payload need not be touched, as a record header written by the producer.

CPU profiles of the proxy and the broker, and throughput at the network card.

Do not put transformations into a path whose value is being a conduit.

A fixed stall points to two buffering mechanisms waiting on each other: Nagle's algorithm on the client and delayed ACK on the server, triggered by a request written as two small writes.

A packet capture shows a gap of about 40 ms before the second segment.

Set TCP_NODELAY, write the header and body in one call, and batch explicitly in the application. Check that the framework has not turned TCP_NODELAY off.

Whenever you add buffering in one layer, ask what the layer below already does.

Block framing: records are length-prefixed inside blocks, and each block starts with a sync marker so it can be read on its own (as in Avro container files). Use newline-delimited records instead if interoperability matters more than efficiency.

Enforce a maximum frame size, validate at ingestion, compress per block rather than per record, and size blocks for the medium, for example tens of MB.

Text is splittable and easy to debug, but slow to parse and prone to delimiter collisions. Binary is compact and fails early, but needs the block and sync-marker design.

Compare counts at every hop (producer, broker per partition, consumer, sink) and find the first hop where they diverge.

If the missing events all sit in one partition, suspect a stalled partition or a hot key. If they are spread out, suspect a routing drop (a record that matched no route), a dead-letter queue nobody reads, or retention expiring under a lagging consumer.

What does each buffer do when full? What is the deduplication key, and where is it stored? What happens to a record that cannot be routed or processed?

Make every loss chosen, bounded and visible: counters for dropped, dead-lettered and expired records, and alerts on end-to-end reconciliation and on growing age.

Only the last week can be replayed from the topic: retention is the hard limit of recovery.

If raw data exists elsewhere (an object-storage landing zone, the source database), reprocess from there into a scratch table, compare it with the current output, then swap it in atomically (write, audit, publish). Pace the backfill so the replay does not overload anything downstream, disable external effects such as emails in the replay configuration, and apply records by version so replayed data never overwrites newer state.

Retain raw input for at least the longest time a bug can go unnoticed, keep transformations deterministic, version dimension tables, and keep a replay path that is tested.

No. Duplicates caused by a replay arrive weeks after the originals, long after their keys expired.

1. Make the sink idempotent by replacement: overwrite the affected partitions, or swap in a rebuilt table atomically, so replay needs no deduplication store.
2. Rely on natural idempotency: upserts by primary key.
3. Extend the window to cover replay. At 5,000 events/s and 40 bytes per key, 30 days is 5,000 × 86,400 × 30 × 40 B ≈ 518 GB of keys, and correctness then depends on Redis's durability.

Option 1 or 2, keeping the deduplication store for online duplicates only. Remember too that flushing Redis silently resets deduplication.

A chain is only as strong as its weakest hop, and only the endpoints can verify the whole.

• The auto-commit setting.
• Whether concurrent processing commits only the highest offset below which everything has completed.
• A try/catch around the enrichment call that carries on, which is an at-most-once decision.
• Whether the audit write is idempotent, not only the main table.
• Whether the dead-letter queue records what it drops.
• Whether the warehouse load replaces or merges, or simply appends.

For each effect, ask for the key and the store.

Reconcile counts from source to warehouse, track a duplicate-rate metric, and run a crash test that kills the consumer 20 times mid-batch.

The weakest hop is the pipeline's real guarantee. Propose fixes in order of value, starting with idempotent sinks.

8% of load on a shard sized for 1/64 ≈ 1.6% is 5 times its share. More servers cannot help: the key cannot be divided.

Partial aggregation; salting into 10 pieces for sums and counts (8% becomes 0.8% per piece); a broadcast join where one side is small; isolating the heavy hitter; and splitting adaptively at run time, since the distribution changes from day to day.

Associativity, memory, or an extra code path.