Scenarios

At this scale, some failure is certain. Design for it.

Re-run failed tasks, for example up to three attempts. Blacklist bad nodes, with a decay time and a quota (at most about 10% of nodes), so failures stay independent rather than repeating on the same machine. Keep shuffle files, so a lost partition does not force re-running a whole stage.

Blacklisting too many nodes shrinks capacity.

Its spare capacity is 10% of maximum throughput, a ninth of the arrival rate, so catching up takes about 3 h × 9 = 27 hours.

To recover from a 3-hour outage in 1 hour, capacity must be (3 + 1) × the arrival rate, so the pipeline should run at about 25% utilization at peak.

Utilization should be set by the recovery budget, not the latency budget. The cost is more machines.

P(real failure) × downtime saved, against P(false alarm) × cost of an unneeded failover. With one real failure a year saving 20 minutes, and four false alarms a year each costing 180 minutes, manual failover wins.

Keep automatic failover for stateless services and read replicas.

Enforce fencing at the storage layer with an epoch number, and expect to lose data equal to the replication lag at the worst moment.

Look for non-determinism: calls to now(), random numbers, lookup tables that have since changed, iteration over unordered collections.

Carry event time through the log instead of reading the clock, and version reference data by date.

Add a test that runs the job twice on the same input and compares output hashes.

The database write can be rolled back, or protected with a write-ahead intent. The card charge is visible to the outside world and cannot be undone, only compensated by a refund.

Restructure so the single irreversible act comes last. Record the intent first, then act, and run a recovery scan that reconciles intents left unfinished by a crash.

Use the order ID as the idempotency key, generated by the client and sent on the first attempt and every retry. The provider, or your own table, stores key → outcome, committed in the same transaction as the charge, and returns the stored outcome on a repeat.

If the provider keeps keys for 24 hours and your longest recovery has been 3 hours, the bet holds with margin. Write it in the runbook.

Keep your own table keyed by order ID, and add a reconciliation job against the provider's records.

Deduplication state grows without bound, hence the window: at 100,000 keys per second with 32-byte entries, one hour is about 11 GB.

The consumer's position lives in the broker and the effect lives in the database, so no order of committing the two is correct.

Store the offset in a database row, updated in the same transaction as the effect, and on startup seek to the stored offset. Fence writes by consumer generation, so a stale owner after a rebalance cannot write.

The consumer's position is now tied to one sink, and rebalances can race.

Use deterministic idempotency keys instead.

The engine's guarantee covers only its own state and its transactional sinks. Sending an SMS from user code is a side effect outside every checkpoint, so it repeats when the job recovers.

Send through an outbox with an idempotency key, if the SMS provider supports one. If not, accept at-least-once and say so; or make the message harmless to repeat ("your results are ready" twice).

With the business owner.

The sink's timeout for prepared transactions, say 15 minutes, was shorter than failure detection + restart + draining the backlog. The sink aborted the prepared transaction before the engine could commit it, and the data in it was lost.

Raise the timeout above the worst-case recovery time, and alert on the age of pending transactions.

Latency through such a sink is quantized to the commit interval.

Keep the retry count in the message headers; a counter in memory resets on restart.

A parse error is permanent; a refused connection is transient. That way a downstream outage does not quarantine good data.

For a ledger, stop the line and page someone. For a clickstream, skip the message to a dead-letter queue.

Reconcile counts, or alert on dead-letter queue depth, with an owner.

It saves a third of the storage (2× instead of 3×) and gives the odd vote. But losing one site leaves a single copy for the hours of rebuild, and the witness must be in a third site, not inside either data site.

Three full copies for critical data, such as patient records; the witness design for less critical data.

State and input positions were most likely restored from different moments, or offsets are committed by a different mechanism from the state.

Checkpoint state and input positions together, with a barrier-based snapshot. Use aligned barriers if checkpoint size is the concern and unaligned if checkpoint time under backpressure is, and note that aligned checkpoints slow down exactly when the job is stressed. Use incremental checkpoints for large state, planning retention, since each checkpoint depends on the previous one.

Use idempotent upserts, or a two-phase-commit sink whose transaction timeout exceeds the checkpoint interval.

Crash tests, and comparing the totals with a batch recomputation.

Exactly-once is impossible for an external effect that cannot be deduplicated. Choose with the business owner: a duplicate letter is annoying and costs money; a missing one may be a legal breach. So choose at-least-once, with tight control of duplicates.

The consumer upserts a row in pending_letters keyed by event_id, an idempotent intermediary. A separate sender records its intent ("about to send"), calls the vendor, then marks the row sent. Make the letter the last effect in the sequence.

Rows left in "about to send" are known unknowns. Reconcile them against the vendor's report, or with a human check, before sending again. Monitor how many such rows exist.

An extra table and a small manual process, in exchange for ambiguity that is visible and bounded.