Skip to content
About
Main content

What physical organization forbids

What can no storage arrangement do, and what does the whole mechanism reduce to?

The central result: two principles are one

The batch-size rule says that a gives a natural batch size: fixed cost ÷ per-byte cost = L ÷ (1/B) = .

Little's rule says that bytes in flight = throughput × latency = .

These are the same quantity.

  • Batch it, and you pay the once per large request.
  • Pipeline it, and you pay it once per outstanding request.
  • Either way, you hold L × B bytes somewhere.

That also explains the second half of the batch-size rule without further argument. When the layer below gets faster, L falls and B rises, and the product moves in whichever direction the hardware moved. For DRAM it stayed near 4 for decades. For storage it climbed from about 2 to about 40 MB when the medium became a network service and not a device.

What this whole mechanism forbids

  • You cannot make all queries fast, only choose which.
  • You cannot have optimal read, write and space at once.
  • You cannot preserve proximity in more than one dimension.
  • You cannot reclaim space without establishing that nobody is reading it.
  • You cannot have both stable identity and useful physical ordering.

The dial of this mechanism

: how much the data is allowed to change after it is written.

  • Layout assumes stability, to buy and vectorization.
  • assumes stability, to buy skipping.
  • Mutation is the tax levied when the assumption is false.
  • Media sets the tax rate.

Everything here is positioned by how much the data is allowed to change after it lands.

What this mechanism did not touch

Every mechanism here assumed a single node and a single writer. Nothing in these five parts required agreement between two machines. Access paths beyond clustering (the index families) remain outside this mechanism. Their absence is now conspicuous, since the clustering synthesis names them as the only answer for high-dimensional filtering. Both of these are the boundary of this mechanism.

Kavya's final design table

QuestionParcelo's answerWhere it came from
File and row-group size~128 MB (3,125 files); inner pages for skippingThe batch-size rule: L × B for , with a 10% latency budget (Data at rest and the one ratio, Physical clustering and Access and media)
Layout of the lake / , with typed ; notes_json shredded and checked is exclusive; the medium forces PAX (Layout models)
Tracking pageKey-value or row copy keyed by parcel_id, with a stated Point reads of whole entities need entity adjacency; the lake has a 200 ms planning floor (Layout models and Access and media)
Corrections and deletesDelta files () and vectors; every τ* (about 20 minutes for hot files)U/d = 640,000× disqualifies (Mutation strategies)
Clustering by (depot_id, scan_time), with C_a = 500 ≤ U = 3,125; reclustering after backfills, budgeted as rent; monitoredPhysical clustering
ReadingRange GETs of ~40 MB each, many in flightL × B for object storage (Access and media)
CostLarge files, few requestsThe fixed cost is also denominated in dollars (Access and media)

Kavya's lesson, in her own words: "Every number I chose was a latency multiplied by a , or a fixed cost divided by a rate. When I changed the place where the bytes sit, every number changed with it. Now I measure the three numbers first."