What physical organization forbids
What can no storage arrangement do, and what does the whole mechanism reduce to?
The central result: two principles are one
The batch-size rule says that a fixed cost per operation gives a natural batch size: fixed cost ÷ per-byte cost = L ÷ (1/B) = L × B.
Little's rule says that bytes in flight = throughput × latency = B × L.
These are the same quantity.
- Batch it, and you pay the latency once per large request.
- Pipeline it, and you pay it once per outstanding request.
- Either way, you hold
L × Bbytes somewhere.
That also explains the second half of the batch-size rule without further argument. When the layer below gets faster, L falls and B rises, and the product moves in whichever direction the hardware moved. For DRAM it stayed near 4 KB for decades. For storage it climbed from about 2 MB to about 40 MB when the medium became a network service and not a device.
What this whole mechanism forbids
- You cannot make all queries fast, only choose which.
- You cannot have optimal read, write and space at once.
- You cannot preserve proximity in more than one dimension.
- You cannot reclaim space without establishing that nobody is reading it.
- You cannot have both stable identity and useful physical ordering.
The dial of this mechanism
The dial is stability versus mutability: how much the data is allowed to change after it is written.
- Layout assumes stability, to buy encoding and vectorization.
- Clustering assumes stability, to buy skipping.
- Mutation is the tax levied when the assumption is false.
- Media sets the tax rate.
Everything here is positioned by how much the data is allowed to change after it lands.
What this mechanism did not touch
Every mechanism here assumed a single node and a single writer. Nothing in these five parts required agreement between two machines. Access paths beyond clustering (the index families) remain outside this mechanism. Their absence is now conspicuous, since the clustering synthesis names them as the only answer for high-dimensional filtering. Both of these are the boundary of this mechanism.
Kavya's final design table
| Question | Parcelo's answer | Where it came from |
|---|---|---|
| File and row-group size | ~128 MB (3,125 files); inner pages for skipping | The batch-size rule: L × B for object storage, with a 10% latency budget (Data at rest and the one ratio, Physical clustering and Access and media) |
| Layout of the lake | PAX / Parquet, with typed columns; notes_json shredded and checked | Adjacency is exclusive; the medium forces PAX (Layout models) |
| Tracking page | Key-value or row copy keyed by parcel_id, with a stated freshness gap | Point reads of whole entities need entity adjacency; the lake has a 200 ms planning floor (Layout models and Access and media) |
| Corrections and deletes | Delta files (merge-on-read) and positional delete vectors; compaction every τ* (about 20 minutes for hot files) | U/d = 640,000× disqualifies copy-on-write (Mutation strategies) |
| Clustering | Global sort by (depot_id, scan_time), with C_a = 500 ≤ U = 3,125; reclustering after backfills, budgeted as rent; clustering depth monitored | Physical clustering |
| Reading | Range GETs of ~40 MB each, many in flight | L × B for object storage (Access and media) |
| Cost | Large files, few requests | The fixed cost is also denominated in dollars (Access and media) |
Kavya's lesson, in her own words: "Every number I chose was a latency multiplied by a bandwidth, or a fixed cost divided by a rate. When I changed the place where the bytes sit, every number changed with it. Now I measure the three numbers first."