Metastore mechanisms

When can a metastore's partition list be silently wrong?

Why a metastore exists

Object stores charge per request and impose a minimum delay per request. Finding the files of a table by listing folders costs O(files) requests, each paying that minimum delay. The cure: precompute the file list and keep it somewhere you can query. That is the metastore. It exists first for an economic reason (request prices) and second for a performance reason.

A partition is a proof

A partition is not a folder. A partition is a predicate (a true/false condition) that is proven true for a set of files, without reading them. For example, "every row in these files has date = 2025-03-01."

Partition pruning (skipping partitions that cannot match a query's filter) works because the metastore holds this claim. The value of the claim is exactly the fraction of files it lets you skip.

SELECT sum(amount) FROM orders WHERE date = '2025-03-01';
-- With the claim "folder date=2025-03-01 holds only that date",
-- the engine opens 1 folder out of 1,095 (three years of days).

The one case of load-bearing metadata that can be silently wrong

In the classic folder-encoded scheme (called Hive-style partitioning) the partition value often is not stored inside the file at all. It lives only in the path:

orders/date=2025-03-01/part-0001.parquet   <- the date is only in the folder name

So there is no way to detect a disagreement between the claim ("these rows are for March 1") and the contents. A file written into the wrong folder gives wrong answers with no error, forever. This design has no verification channel. Manifest-based designs (below) remove this property.

Scale mechanics: the metastore is a database

The metastore is an OLTP database (a database serving many small reads and writes). It is usually the least-engineered database in the stack.

Row count is roughly partitions × (1 + number of statistic columns).
Example: hourly partitions for three years = 24 × 365 × 3 = 26,280 partitions (about 26 thousand rows per table, before statistics). A thousand such tables = about 26 million rows, often stored in a single relational instance.
Adding a partition is a write. Planning a scan is a read of thousands of rows. So planning time grows with the number of partitions.
High-frequency ingestion turns the metastore into a write-contention hot spot (many writers fighting over the same rows) exactly when ingestion is busiest.

Little's Law governs the ceiling. Little's Law says: average number of requests inside a system = arrival rate × average time each spends inside. Applied here: metastore request rate × planning time = number of planning slots in use at once.

The small-file problem is first a metadata problem

The small-file problem is usually described as a scan-efficiency problem. It is first a metadata problem. One file written means one metadata write, and the fixed cost of that metadata write is far greater than the marginal cost of the file. A mismatch between the size of the unit of work and the size of the machinery amplifies harder in the catalog than in storage, so the metastore falls over before scans get slow. This is Hana's problem 3.

Partition granularity has a metadata price

Each partition entry has a fixed metadata cost, so finer partitions buy better pruning with more catalog rows, more planning reads and more small files. How fine to partition is a layout decision (a partition should hold a handful of files of the size the object store prefers); the metastore is where an over-fine choice is paid first.

A partition is a proof about files you have not read. The metastore is an OLTP database sized by object count, not by bytes.