Scenarios

One connection to S3 delivers about 80 MB/s, so 50 TB ÷ 80 MB/s = 625,000 connection-seconds. In 300 seconds that needs about 2,100 parallel streams and about 170 GB/s of total network bandwidth (2,100 × 80 MB/s).

One machine cannot receive 170 GB/s: a 100 Gb/s network card gives 12.5 GB/s. So at least 14 machines are needed for the network alone; with decompression and memory bandwidth added, more like 30–50.

About 40 nodes, each running 50–100 concurrent range GETs of 8–64 MB, spread across many key prefixes to stay under per-prefix request limits (which return 503 errors). Read only the needed columns, so fewer bytes cross the network at all.

Cost (per-request pricing and egress), time for S3 to split busy prefixes, and stragglers: hedge slow requests by sending a duplicate.

Feasible, but it is a multi-node network-bandwidth problem, not a storage problem.

The chance that a request meets no slow shard (none slower than 50 ms) is 0.99⁴⁰ ≈ 67%, so 33% of requests are slow, not 1%. The service's p99 is about the shards' p99.975, far above 50 ms. The SLO will very likely be missed.

1. Reduce fan-out: re-partition so a typical request touches about 8 shards (0.99⁸ ≈ 92% fast).
2. Hedge: if a shard has not answered after 15–20 ms, send the request to a replica. A few percent more load buys a large tail improvement.
3. Shrink each shard's own tail: find what makes shards slow, such as GC pauses, compaction or a noisy neighbour. Slowness that hits many shards at once is worse than the formula assumes.
4. Return partial results at a deadline, if the product allows it.

Hedging adds load. At high utilization it can push the system past the knee of the queueing curve, so cap hedges at about 5% of requests and watch utilization.

Write amplification has grown. Over time free blocks fragment, the drive copies more live pages during garbage collection (WAF = 1 / (1 − v), where v is the fraction of each erased block still live), and that background work steals bandwidth. Test: compare NAND writes with host writes over time, using smartctl.

TRIM is not reaching the drive: a mount option, a virtualization layer or a RAID controller blocks it, so the drive believes it is nearly full. Test: check discard / fstrim, then measure again after running fstrim.

The application mixes data of different ages in place, for example random updates in a B-tree-like layout, which drives amplification up.

Over-provision more (leave 20–30% of the drive unpartitioned), run TRIM regularly, separate hot and cold data, or move to an LSM or another sequential write layout.

By R = S / (1 − ρ), with S the service time and ρ the utilization: at ρ = 0.6, R = 2.5 S; at ρ = 0.85, R = 6.7 S on average, and the tail is much longer.

New servers take minutes to start. During those minutes the queue grows, clients time out and retry, and the arrival rate rises further. Scaling at 85% reacts after the knee of the curve, when the feedback loop has already started.

1. Scale at 50–60%, or on a leading indicator such as queue depth or concurrent requests (by Little's Law, L = λW, concurrency rises before CPU does).
2. Reduce variation in service time, for example by moving slow endpoints to their own pool; by Kingman's formula, variability multiplies the wait.
3. Add admission control and load shedding at the edge.
4. Give retries a budget and add jitter.

Batch workers, where no one waits on a single request, can safely run at 95%.

S3 has no atomic update of several objects and no atomic rename; consistency is per object.

Writers first upload their new data files. These are invisible until something references them, so uploading is safe to repeat. Each writer then writes a new metadata file describing the new table state, and commits with an atomic compare-and-swap on one pointer: a conditional write of a "current version" object, or a row in a catalog or a table such as DynamoDB.

If two writers race, one loses the swap. It re-reads the winner's metadata, checks whether they touched the same files or partitions, and retries if not.

Files left by failed writers are removed later by garbage collection. This is the general pattern for immutable storage: write new, switch one pointer, collect the old.

Retries must be idempotent. Garbage collection must not delete files that an in-flight writer has uploaded but not yet committed, so give it a grace period.

A round trip between Mumbai and Virginia is about 200 ms, so any commit that needs both regions takes at least 200 ms.

1. A home region for each account, with its leader there: local users are fast, remote users pay about 200 ms per write, and reads can be served locally if slightly stale reads are acceptable.
2. Synchronous consensus across regions (Spanner-style): strong consistency everywhere, at the cost of at least one inter-region round trip per commit. Acceptable only for low write rates, or if 100–300 ms writes are tolerable.
3. Asynchronous replication with conflict resolution: fast everywhere but not strongly consistent, which is usually wrong for balances.

Pin each account to a home region by its user's location, keep the quorum across availability zones inside that region (1–2 ms apart), and replicate asynchronously to the other region for disaster recovery. Be explicit that strong consistency, low latency and active writes in several regions cannot all be had at once.

The symptoms point to incast, not bandwidth: low average utilization, a sharp jump in p999, and a clear link to the number of workers. Check switch-port drop counters, TCP retransmissions and timeouts (ss -ti, nstat), and whether the p999 values cluster near the retransmission timeout (200 ms, 1 s).

1. Add a few milliseconds of random jitter to worker replies.
2. Reduce fan-out (fewer, larger partitions), or aggregate in two levels so no single receiver hears from 150 senders.
3. Enable ECN-based congestion control (DCTCP) and lower tcp_rto_min.
4. Limit replies in flight with credits: the coordinator grants permission to send.

By the Universal Scalability Law, past a certain number of workers each added worker reduces total throughput.

Where is the time spent? A cache helps only if the same data is read repeatedly, and only if the slow path it avoids is the dominant cost.

With hit ratio h, average latency = h × t_cache + (1 − h) × t_backend. With a 100 ms backend, a 10× gain needs h of at least 90%; at h = 50% the best possible gain is 2×.

Tail latency is set by the misses, so p99 may not improve at all.

The cost moves rather than disappears: to memory, to invalidation complexity and stale reads. And if the cache starts empty or fails, the backend receives its full load, possibly while already near the knee of its queueing curve.

Measure the hit ratio on a replay of real traffic before building anything.

Cleaning cost is u / (1 − u) bytes moved per byte freed, where u is utilization: 2.3 at 70%, 9 at 90%, 19 at 95%.

Choose the target utilization from your write-amplification budget and the drive's endurance, not from capacity: headroom is write amplification bought in advance.

The trend: load growth pushes utilization towards the cliff.