Latency numbers and back-of-the-envelope estimates
Which latency ratios should you carry in your head, and how do you check that a design can work before building it?
Latency numbers by order of magnitude
The point of this table is not to memorize it, but to see ratios. Remember one number and derive the others.
| Operation | Latency | Relative (1 cycle = 1) | Human scale (1 cycle = 1 s) |
|---|---|---|---|
| Register / 1 cycle | 0.3 ns | 1 | 1 second |
| L1 cache | 1 ns | 3 | 3 seconds |
| Branch mispredict | 3–5 ns | ~13 | 13 seconds |
| L2 cache | 4 ns | 13 | 13 seconds |
| L3 cache | 15–20 ns | 60 | 1 minute |
| DRAM | 80–100 ns | 300 | 5 minutes |
| Remote NUMA node | 150–200 ns | 600 | 10 minutes |
| Context switch (effective) | 10–50 µs | ~100,000 | 1 day |
| NVMe read | 80–100 µs | 300,000 | 3.5 days |
| Same-rack RTT | 100–200 µs | 500,000 | 6 days |
| Cross-AZ RTT | 0.5–2 ms | 3,000,000 | 5 weeks |
| HDD seek | 8–13 ms | 30,000,000 | 1 year |
| S3 GET (TTFB) | 20–100 ms | 200,000,000 | 6 years |
| Cross-continent RTT | 100–200 ms | 500,000,000 | 16 years |
| Tape mount + position | 30 s – 4 min | ~10¹¹ | 3,000 years |
- Ratios worth remembering: DRAM ≈ 100× L1. NVMe ≈ 1,000× DRAM. Same-rack network ≈ NVMe. S3 ≈ 1,000× NVMe. Cross-region ≈ 1,000× same-rack.
- The human-scale column teaches the most. If a register access takes one second, an S3 GET takes six years. After that, "combine your object-store requests" feels obvious.
- Look at the gaps, not the values. The two largest jumps are DRAM → storage (~1,000×) and same-region → cross-region (~100×). These are exactly where the architecture boundaries of real systems are: memory vs. disk, single-region vs. multi-region. Abstraction boundaries follow latency cliffs. That is not a coincidence.
Back-of-the-envelope estimation
Take this seriously: it is a design tool, not a party trick. An estimate correct within 10× tells you whether an architecture is possible, in five minutes instead of two weeks of prototyping.
Method:
- Find the dominant cost. One term usually dominates; ignore the rest.
- Use round numbers.
- Write units on every number. Units catch most errors automatically.
- Check against a physical limit.
- Give the answer as a range.
Numbers to know by heart:
- 1 day ≈ 10⁵ seconds (86,400). 1 year ≈ 3 × 10⁷ seconds.
- Memory: ~100 ns, ~50 GB/s per socket.
- NVMe: ~100 µs, ~5 GB/s.
- Network: ~100 µs in a rack, ~50 ms across a continent.
- S3: ~50 ms to first byte, ~80 MB/s per connection.
Worked example: scan 10 TB of Parquet from S3
- 10 TB = 10,000,000 MB at 80 MB/s per connection → 10,000,000 ÷ 80 = 125,000 connection-seconds.
- With 100 parallel streams: 125,000 / 100 = 1,250 s ≈ 21 minutes.
- With 1,000 streams: ~2 minutes.
But 1,000 × 80 MB/s = 80 GB/s of incoming data and decompression. One machine cannot do that: its memory bandwidth is ~50–200 GB/s and decompression needs more. So the answer is: this is a multi-node bandwidth problem, and the limit is the cluster's total network, not S3. That took two minutes and changed the whole design.
The real job of an estimate is to disprove. It shows that an approach cannot work before you build it. "This needs 10 Gb/s per node and we have 1" ends the discussion. And when a later measurement differs from the estimate by more than ~10×, you have found something you do not understand, which is the most valuable result possible.