File, row-group and page skipping

How deep should skipping metadata go, and how does skipping differ from an index?

This is where the opening derivation is used, plus one structural tension.

Skipping is hierarchical: file, then row group, then page. Each finer level buys precision with more metadata. Page-level statistics for a one-billion-row table with 10,000 rows per page give 10⁹ / 10⁴ = 100,000 entries per column. For 200 columns that is 20 million entries. That is a dataset in itself, and it must be read before it can save a read. This is why Parquet keeps two things apart: footer statistics at row-group level, which are always read, and an optional page index, stored separately and fetched only when a predicate might benefit. The metadata hierarchy mirrors the data hierarchy for the same reason index hierarchies exist.

Read amplification, in the form you can compute. The skipping unit is a block and the predicate selects rows. The read amplification is

block size / (matching rows per block)

If one matching row lands in each of 1,000 blocks, you read 1,000 blocks to produce 1,000 rows. This is the granularity mismatch (you can only read whole blocks, but you want single rows) written as arithmetic. The useful corollary: making the predicate more selective does not help once you are reading one block per matching row. Beyond that point the only remedy is to change the clustering so that matching rows sit together.

Another observation reframes the mechanism. The block size was not chosen for skipping. Row-group size was set by I/O economics: a fixed cost per request gives a natural transfer size (it is wasteful to read less than a few megabytes per request). That was settled long before anyone thought about zone maps. Data skipping inherits a granularity that was picked for another purpose and is stuck with it. This is a recurring pattern: the unit of data reduction is almost never chosen by the data-reduction mechanism.

Skipping metadata versus an index

Here is the cleanest boundary between skipping and indexing.

• An index is a positive structure: value → location. It proves presence. Its size is proportional to the data. It must be maintained on every write.
• Skipping metadata is a negative structure: location → summary of values. It proves absence. Its size is proportional to the number of blocks. It is written once with the data and needs no maintenance, because the data is immutable.

Skipping is what indexing becomes on an immutable substrate. This is why lakehouse table formats have rich skipping metadata and almost no secondary indexes. It is not a maturity gap. It is the correct design when updating an index in place is not possible.

In one sentence: skipping is a negative index priced in metadata per block and not per row; its read amplification is block size over matching rows per block; and its granularity was inherited from I/O economics, not chosen.