Scenarios

Pauses and network spikes of hundreds of milliseconds are common, so a 500 ms timeout would cause frequent false suspicions and constant changes of leader.

Each election costs availability. If writes carry epochs, it does not cost correctness.

An adaptive detector (phi-accrual style) whose threshold sits above the tail of normal delays; fix the 9-second pause itself, for example by tuning garbage collection; and keep the safety rule independent of any timeout.

The holder paused past the expiry, the lock was granted to someone else, and the first holder then woke and wrote, still believing it held the lock.

A fencing token: each grant comes with an increasing number, and the storage layer rejects writes carrying a lower number than one it has already seen. Expiry based on clocks can never be a safety mechanism on its own.

The storage must be able to check the token; a plain object store may need conditional writes.

Add one node, wait until it has caught up and the new configuration has committed, then add the second.

Going from 3 to 5 in one step can create two majorities that do not overlap: 2 of the old 3 and 3 of the new 5.

Joint consensus, where decisions temporarily need majorities of both the old and the new configuration.

Two sites cannot distinguish a dead peer from a broken link between them, so automatic failover risks both sites promoting themselves: split brain.

Put a small witness, a consensus member that stores no data, in a third failure domain, so a majority of three survives the loss of any one site. Or accept manual failover.

An extra location, and cross-site round trips on every decision.

At a throttle of 100 MB/s: 12,000,000 MB ÷ 100 MB/s = 120,000 s, about 33 hours.

Copy, catch up, cut over, so the data stays readable throughout. Delete the source copy only after the routing map has flipped.

Peak disk use on the sources, since data exists twice during the move, which matters at 80% full. Throttle to protect foreground latency.

1 − 0.99⁵⁰ ≈ 39.5% of page loads include at least one service's p99 event.

• Wasted work: hedge at p95, capped at 5% extra requests; check the hedge rate to confirm the slowness belongs to attempts.
• Weakened semantics: return partial results for approximate parts such as counters, if the page can tolerate a missing input.
• Waiting: wait for the parts that must be exact.

Aggregation trees to fix the tail. And check that choosing large shards, good for fan-out, does not make recovery slow.

Five layers with three retries each make up to 3⁵ = 243 attempts per user action.

• Retry at one layer only: the one closest to the failure that can judge whether a retry is safe.
• A token-bucket retry budget at the client, about 10% of recent successes.
• Absolute deadlines passed along every call, so abandoned work stops.
• Idempotent operations, with request IDs and a deduplication window longer than the longest retry.

Some legitimate retries are refused during a partial outage. Backoff with jitter is a separate, complementary mechanism.

It breaks the rule that control-plane load must grow with changes, not requests, and probably the rule that the control plane must stay off the data path.

Move bulk facts out (into manifests or data-plane metadata) and keep only small, rarely changing facts in consensus: membership, leadership, the partition map. Let the data plane keep working on its last known map if the control plane is down, and protect against stale configuration with fencing.

Name the hidden dependency, and add an alarm and a regular drill for losing the control plane.

Keep the common transaction and the common query inside one shard.

Checkout and history are customer-centric, so shard by customer_id: orders, addresses and payments sit together, with no cross-shard transactions.

Revenue by product touches every shard, and a query across all shards is as slow as the slowest: with 100 shards each slow 1% of the time, about 63% of such queries hit a slow one. Do not run it on the OLTP shards; feed an analytical store, partitioned by product and date, through CDC.

Use 1,024 or more logical shards mapped onto physical nodes, so adding nodes moves logical shards and never changes the key mapping.

A very large customer (key skew) needs isolation, and changing the shard key later is a migration.