The cost of one access
What does one touch of storage cost, and why does that set the size of every unit?
The exchange rate
Three parts chose arrangements: layout, mutation strategy and clustering. This article is not a choice at all. It is the exchange rate in which those arrangements were priced, and it has been setting your constants from underneath the whole time. The floor under the 128 MB row group, the U/d that disqualified copy-on-write, and the min(G, 1/s) payoff cap for clustering: none of those numbers came from the parts that used them. They came from here.
It is not a peer of the other three groups.
The new constraint is one sentence, and everything above quietly assumed the opposite:
The cost is fixed + bytes ÷ bandwidth. The only thing that matters about a medium is the ratio between those two terms. That ratio has a natural expression:
- latency
- — the setup cost paid for each operation, whatever its size
- bandwidth
- — bytes per second once data streams
- characteristic size
- — the size at which half of your time is setup
- Below this size, you pay for latency and the bytes are free.
- Above this size, you pay for bytes and the latency is free.
This one number, which you can measure in an afternoon, determines your minimum efficient request, your batch sizes and your prefetch depth. At the end of this article we see that it is also the answer to two of this mechanism's principles, which turn out to be the same principle.
What stops mattering here is arrangement. This article does not care whether you chose columnar or sorted. It cares about three measured numbers: latency, bandwidth, and how many requests must be in flight. These numbers span six orders of magnitude across the media you use.
The governing question:
The three numbers
| Medium | Latency | Bandwidth | L × B | Requests in flight for full rate |
|---|---|---|---|---|
| DRAM | ~80 ns | ~50 GB/s | 4 KB | ~10-20 (memory-level parallelism, MLP) |
| NVMe SSD | ~80 µs | ~7 GB/s | 560 KB | ~140 × 4 KB |
| HDD | ~10 ms | ~200 MB/s | 2 MB | 1 (the head is serial) |
| Object store | ~50 ms | ~80 MB/s per connection | 4 MB | ~64 × 8 MB for 10 GB/s |
Arithmetic check: DRAM 80 ns × 50 GB/s = 4,000 bytes. NVMe 80 µs × 7 GB/s = 560,000 bytes. HDD 10 ms × 200 MB/s = 2,000,000 bytes. Object store 50 ms × 80 MB/s = 4,000,000 bytes.
The L × B column goes from 4 KB to 4 MB: three orders of magnitude. And that is before applying the rule "latency should be at most 10% of the transfer time". That rule multiplies the object-store figure by 9 (to about 36 MB, roughly 40 MB) and puts it on the same floor that Physical clustering derived from another direction.
(Note on the last column: the figure for the object store counts only the latency. Each 8 MB GET also needs about 100 ms of transfer time at 80 MB/s, so with that included each request lasts about 150 ms and more requests are needed than the table shows. The table gives the order of magnitude.)
DRAM's characteristic size is 4 KB. That is the virtual memory page size, and it is no coincidence. It is the same calculation that hardware designers made in the 1970s, and it is still right. The second half of the batch-size rule (the size follows the hardware) can be seen here: an old hardware decision is still visible in today's numbers.
Why the numbers are what they are
Where these numbers come from is a property of the medium, not of the layout. "Sequential versus random" is really paying the fixed cost once or once per request, and keeping a fast device busy takes many requests in flight at once (queue depth). Who holds the bytes once they arrive (the kernel's page cache, a memory map, or the database's own buffer pool with direct I/O) decides how many requests a reader can keep in flight; that is memory management's question. For layout, two consequences matter. A request smaller than L × B is mostly waiting, so the unit you write must be at least that large. And a reader that must touch many small units is not rescued by a faster device, only by fewer, larger units or by issuing them concurrently.
Alignment and page-sized I/O
This is the mismatch rule in its purest form.
A 4 KB write that straddles two 4 KB device pages forces a read-modify-write of both pages: two reads plus two writes to accomplish one logical write. Misalignment turns a 1× operation into roughly 4×, and it does so silently.
It bites in four places:
O_DIRECTrejects unaligned requests outright, which is at least honest.- SSD page and erase-block alignment decides whether a write is a simple program or a program plus a relocation of other data.
- File system block size against physical sector size. This caused a widespread real regression when "512e" drives reported 512-byte sectors while physically using 4096-byte sectors. Partitions created with the old assumption ran misaligned for years.
- At the object level, row-group boundaries that do not align with range-request or multipart-upload boundaries turn one GET into two.
The general point:
Boundary effects are noticed too rarely because they never show up in an average, only in the specific case, and in the specific case the loss is a fixed multiplier, not a tail.