Disk and flash
What do a spinning disk and a flash chip each force on the software above them?
Rotational disk (HDD)
How it works. A hard disk is a stack of spinning metal platters with a magnetic coating. A head on a moving arm floats above the surface. To read data, the disk must (1) seek: move the arm to the right ring (track), then (2) wait while the platter rotates until the right spot passes under the head. Both steps are mechanical, so slow: 8–13 ms per random access.
Position on the dial: high barrier (magnetism needs no power). One arm serves ~20 TB, so the granularity is coarse and the fixed cost is mechanical.
What matters:
- Mechanical cost never followed Moore's law (chip speed doubling every two years). Random reads per second on a disk have stayed near 100 since the early 1990s, while capacity grew about 1,000×. Reads per second per terabyte fell by a factor of 1,000. This is why the industry stopped designing random-access systems for disks and moved to sequential ones.
- Positioning cost × transfer rate ≈ 1.3 MB. Example: positioning takes ~13 ms, then the disk streams at ~100 MB/s. 0.013 s × 100 MB/s = 1.3 MB. If you read less than ~1.3 MB per positioning, you mostly pay for arm movement. Every question like "why are HDFS blocks 128 MB?" ends here.
- Sequential and random access are not two different patterns. They are paying the fixed cost once, or paying it for every request.
- SMR (Shingled Magnetic Recording) shows Cause 2 clearly. SMR overlaps tracks like roof tiles to get ~20% more density. The price: you cannot rewrite one track without damaging its neighbor. So the disk is split into zones that can only be written in order, with background reorganization. The disk reinvented garbage collection as soon as it became cheaper per bit.
NAND flash (SSDs)
How it works. Flash stores a bit as electric charge trapped in a cell. Cells are grouped into:
- a page (4–16 KB): the unit you read and write;
- a block (4–24 MB, hundreds of pages): the unit you erase.
| Operation | Unit | Notes |
|---|---|---|
| Read | page | fast (tens of µs) |
| Program (write) | page | only into an erased page, in order inside the block |
| Erase | whole block | slow (~ms) and wears the cells |
Position on the dial: non-volatile thanks to a high barrier, but writing means tunneling through that barrier, which damages it. The endurance limit is the direct price of retention.
The sentence that explains the whole design
Flash cannot overwrite a page. To change data in place you would erase the whole multi-megabyte block and destroy every other page in it. So nobody does that.
The FTL and garbage collection
Because of this granularity mismatch (Cause 2), flash has no in-place update. So every SSD contains an indirection layer called the FTL (Flash Translation Layer): a table that maps "the address the operating system thinks it writes to" to "whichever physical page is free". When you "overwrite" data, the SSD writes the new version to a fresh page, updates the table, and marks the old page invalid.
Invalid pages collect inside blocks that also hold valid pages. When free blocks run low, the drive runs garbage collection (GC): choose a block, copy its still-valid pages elsewhere, erase the block. The FTL and GC are not a vendor's choice. Physics forces them. (This is the "immutable storage → indirection → garbage collector" pattern of Cause 5.)
Write amplification (WAF)
WAF = bytes physically written to flash ÷ bytes the host asked to write. The pages copied by GC are extra writes. WAF of 10 means each 1 GB you write costs 10 GB of wear.
Many people think WAF depends mainly on how full the drive is. That is only a rough proxy. WAF depends on lifetime clustering: whether data written together also dies together.
If data that dies together was written together, whole blocks become invalid at once and GC copies nothing. If lifetimes are mixed (long-lived and short-lived data in the same blocks), every block keeps a live part, and you copy live pages again and again (WAF 10–20×). Sequential writes are good not because they are sequential, but because their data has the same age.
Other facts that follow
- Over-provisioning buys a lower WAF with space (Cause 2's trade). The drive hides some capacity, so GC can find emptier blocks. Enterprise drives reserve ~28%, consumer drives ~7%.
- TRIM solves an information problem. When you delete a file, the filesystem knows the space is free but the SSD does not, so the FTL still thinks every address ever written is alive. The drive looks nearly full and WAF rises. The TRIM command tells the drive which addresses are no longer used. Nothing physical changed; the device only lacked knowledge.
- Endurance is a budget you spend. Usable writes = rated cycles ÷ WAF. Your architecture sets the denominator.
- Read disturb: reading a page slightly stresses its neighbors, so heavily read blocks must eventually be rewritten. Reads eventually cause writes.
Stacked logs multiply
The FTL is a log. Put a log-structured store or a journaling filesystem on top of it and you have logs stacked on logs. Each has its own segment lifetime, its own garbage collector, and its own idea of what "sequential write" means. Each layer believes it is optimizing. Together they multiply write amplification and make latency unstable, because the lower layer's GC is triggered by the upper log's rotation. The reference is the 2014 paper Don't Stack Your Log On My Log (Yang et al.). F2FS and other log-structured filesystems exist partly to merge two of the logs.
The general lesson: identical mechanisms at adjacent layers do not add up. They interfere. The lower layer's batching is defeated by the upper layer's batching. Expect the same with caches, retries and batching.
Why LSM-trees exist
An LSM-tree (Log-Structured Merge tree) is the write-optimized structure used by RocksDB, Cassandra, LevelDB and others. This is the physical reason it was invented.
- A B-tree (the classic index) updates records in place. On flash the problem is not the tree shape. A 100-byte update causes a full-page write at a random address, so blocks fill with data of mixed age and die one page at a time. That is granularity mismatch and lifetime mixing together.
- LSM's move: collect writes in memory (a memtable), then write them as large, sorted, never-modified files (SSTables) holding data of the same age. It does not remove amplification. It turns many small scattered writes into large same-age writes, so the device WAF falls to ~1.
- It then pays the cost back in software. Background compaction (merging and rewriting files) can reach 30–40× write amplification with leveled compaction. Cost conserved, but moved (Rule 1). The only design question is which layer is cheapest to pay in.
- Tiered compaction, leveled compaction and B-trees are three corners of the RUM triangle (Cause 4). Bloom filters reduce read cost by spending memory. There is no fourth option that wins all three.
Read/write asymmetry and endurance
- Reading samples a state. Writing forces it over a barrier. Sampling is cheap and does not damage anything. Forcing is slow, uses energy and causes wear.
- So caching and replication cost different amounts. Adding read replicas (extra copies serving reads) is almost free for storage. Adding write paths multiplies the expensive side, because every copy must do every write.