Layout models: row, column and PAX

Should a row, a column or a group of rows be the unit that sits together?

A row is a logical fiction

Every other part of this mechanism takes the internal shape of a record as given. Mutation asks what happens when a value changes. Clustering asks which unit a record goes into. Media asks what a touch costs. Layout is the only decision that chooses whether a row exists as a physical object at all.

That makes it the least reversible decision here.

• Changing a sort key costs a recluster (a rewrite in a new order).
• Changing a mutation strategy costs a rewrite of the write path.
• Changing from rows to columns changes the file format, the compression, the code of every operator, the in-memory representation inside the engine, the format used between machines, and the cost model of the query planner.

Layout has the largest blast radius and the earliest deadline.

The constraint behind it is very simple, and it cannot be bypassed:

The relational model says a row is a unit, but nothing physical requires that. Two values can be neighbours because they describe the same entity (same parcel), or because they come from the same domain (both are weights). A value cannot be adjacent to both its row-mates and its column-mates at the same time.

The governing question for this article: for the reads you expect, what is the smallest contiguous byte range that answers one read, and does that range correspond to an entity or to an attribute?

---

The core lattice: four numbers per layout

Take a table with n columns and a mean value width of w bytes. A row is r = n·w bytes. The storage access unit is B bytes. A query touches k of the n columns, over m of the N rows. The costs then follow mechanically:

The names: NSM (N-ary Storage Model) means row storage. DSM (Decomposition Storage Model) means column storage. PAX (Partition Attributes Across) is the hybrid, explained below.

The point-read column is the one people underestimate.

That single ratio explains the entire shape of the storage industry.

The summary line. Adjacency is a zero-sum allocation. The layout question is which of the two amplifications you would rather pay: n/k (rows) or B/w (columns).

---

Row-oriented layout (NSM)

This layout takes the logical model literally. The reasoning from first principles: if the unit of update is the entity, and the unit of read is the entity, then storing the entity contiguously makes the write unit, the read unit and the access unit the same. The mismatch rule says amplification comes from mismatch between exactly these units. So NSM's virtue is not speed. Its virtue is that it creates the fewest mismatches for entity-shaped access.

What it forbids: reading one attribute cheaply. The amplification n/k is not a constant factor you can engineer away. It is the ratio of what you stored beside the value to what you wanted. A 100-column table answering a 3-column query wastes 97% of every byte moved, at every level: disk to page cache, page cache to CPU cache, CPU cache to register. The waste at the cache-line level (the 64-byte unit in which a CPU moves memory) is independent of the waste at the block level, so the two compound. It is one mismatch counted at two granularities.

A common statement needs correction. "Row stores are for OLTP" (online transaction processing: many small reads and writes) is a true correlation with a wrong mechanism. The real driver is not transactions. It is two ratios: how many columns you touch per row you touch, and how large an update is compared with the row width. An analytics system that mostly does point lookups wants rows. A transactional system that rewrites whole rows on every update does not care about layout on the write side. Diagnose by access shape, never by workload label.

Summary. Row layout minimizes mismatch when the entity is the unit of both read and write. When it is not, it wastes 1 − k/n of every byte.

---

Columnar layout (DSM)

Store each attribute as its own sequence. The advertised benefit is that the n/k term collapses to 1. That is the smaller half of the story. Repeating only that half undercounts the advantage of columnar storage by roughly an order of magnitude.

Effect 1: projection

Read only the columns you need. This is the n/k saving.

Effect 2: entropy (compression)

The real mechanism is entropy, a measure of how unpredictable data is. In a row, adjacent bytes are a timestamp, then a UTF-8 string, then a float. The correlation between neighbouring bytes is near zero, so a generic compressor finds few repeats in its window and gives only 2-3×. In a column, adjacent values are draws from one distribution. That allows type-aware encodings instead of generic ones: dictionary codes (a column with k distinct values needs ⌈log₂ k⌉ bits per value, so about 200 countries fit in 8 bits instead of 12 × 8 = 96 bits), deltas, run lengths and bit-packing, often 5-20× on real columns, and sometimes 100×. How each one works is encoding's subject. What matters here is that they are only possible because like values sit next to each other. RLE, for example, is free when the column is sorted or has few distinct values, so clustering decisions (Physical clustering) feed back into the compression ratio. The two parts are not independent, and that coupling is worth remembering.

These effects multiply with the projection benefit. They do not add to it. The effective reduction is

where c is the compression ratio. That product, not the projection term alone, is why columnar storage won analytics decisively and not just marginally.

Effect 3: vectorization

On modern hardware this third effect is often the largest. A column chunk is a typed array: contiguous values of the same width. Operators run over it without per-row branching and without following pointers. This is what makes vectorized execution and SIMD (one CPU instruction working on many values at once) possible at all. The effect has nothing to do with disk I/O. It survives even when all data is already in memory. It is the same effect as the well-known result that a loop over sorted data runs faster than the same loop over shuffled data: the CPU's pipeline (the assembly line inside the CPU) stops stalling when the data is predictable and uniform, because it can guess the next branch correctly. Predictable, uniform data is cheap not because there is less of it, but because the pipeline does not wait. On NVMe this term is frequently larger than the I/O term.

The cost: the encoding block becomes the unit of update

Encoding fuses values together. A dictionary-encoded, bit-packed page cannot have one value changed without decoding and re-encoding the whole page. Compression converts the unit of update from a value into a block.

So the layout decision made for read efficiency has produced a new granularity mismatch on the write side. This is the mismatch rule again, caused by your own optimization. This is why columnar systems do not update in place. It is not a preference. The cost structure forbids it, and these systems hand the problem to merge-on-read, delete vectors and copy-on-write (all explained in Mutation strategies). Columnar layout does not merely coexist with immutability. It forces it. That is the handover to Mutation strategies.

Summary. Columnar layout buys three effects (projection, entropy, vectorization) and pays by making the encoding block the new unit of update.

---

PAX (hybrid) layout

Why does PAX exist? Because in pure column storage the column chunks sit in separate places. The distance between a row's first and last field has no upper bound, and the cost of tuple reconstruction (putting a row back together from its columns) grows with that distance. PAX bounds it: split the rows into groups of G rows, and store column-major inside each group.

G is not a taste parameter. It is fixed cost divided by per-byte cost, computed for the substrate (the batch-size rule).

• On a local disk with sub-millisecond seeks, G lands around a few MB.
• On object storage, a GET request carries 20-100 ms of latency plus a per-request fee. The fixed cost is thousands of times higher, so G rises with it. Parquet's ~128 MB row group and ORC's 64 MB stripe are that division carried out. They are not conventions.

One mechanism, three reinventions

PAX was invented in 2001 (Ailamaki et al.) to fix cache-line waste. A row-store scan of one attribute pulled whole rows into the CPU cache and used only a fraction of each 64-byte line. The identical mechanism was reused about a decade later at disk-block size, and again at object size. That is the second half of the batch-size rule in its cleanest form: the fixed cost moved upward as the layer below got faster, and the same design followed it up. One mechanism, three granularities, three independent reinventions.

The hidden cost of PAX

A point read still touches n locations, now close together, and pays a decode cost for each one. And the row group is the unit of independent metadata and of rewrite, so G sets a floor on the cost of changing anything. Deleting one row in a 128 MB row group forces either a 128 MB rewrite or a tombstone (a "this is deleted" marker). A larger G amortizes reads better and makes writes coarser. You cannot tune both directions.

Summary. PAX bounds the reconstruction distance to the row group. The row group size is the fixed cost of the substrate divided out, and it is also the floor of your rewrite granularity.

---

Wide-column (column-family) layout

Structurally this is PAX with the grouping fixed at table-creation time (DDL) and not derived when data is written, and with family boundaries in place of row-group boundaries. Inside a family, storage is row-major over a sorted key.

The trade-off follows directly. If the system derives the grouping from the data, it needs no foresight. If you declare it in the schema, you place the bet earlier and with less information: you must predict at design time which columns are read together. What you receive for this is per-family storage policy: independent compression, TTL (time-to-live: automatic expiry), in-memory flags and separate files. PAX cannot offer this, because its groups are anonymous.

A correction, because the name misleads. Cassandra and HBase are often described as "column-oriented". They are not columnar in the DSM sense. A partition is stored contiguously, row by row, as a sorted map of cells. A "wide row" is many cells under one key, not a column chunk. These systems are key-value stores with structured, sorted composite keys. If you reason about their scan performance with columnar intuition, you will be wrong by the n/k factor, in the bad direction.

Summary. Column families are PAX with the groups named at schema time. You pay with foresight and receive per-group storage policy.