Glossary

62 terms, in the sense this mechanism uses them. The platform glossary has every term across all mechanisms.

Algebraic statistic: A statistic (count, sum, min, max) that can be merged exactly from per-file values.
Amortization: Spreading a fixed cost over many units so the cost per unit falls.Paying a fixed cost once and spreading it over many units. One delivery truck for 1,000 parcels costs less per parcel than 1,000 bicycles.
Amplification: Extra bytes read, written or stored compared with what was logically needed.If you ask for 200 bytes and the system must move 4,096 bytes, the amplification is about 20. Read amplification means extra bytes read. Write amplification means extra bytes written. Space amplification means extra bytes stored. It is the hidden cost behind most slow systems.
Anti-join: A join that keeps left rows with no match on the right.
Auto-compaction: Compaction that starts by itself from triggers, without a person scheduling it.
Back-of-envelope units: 1 TB = 1,000,000 MB. 1 GB = 1,000 MB. A millisecond (ms) is 1/1,000 of a second.
Bin-packing: Choosing groups of small files to rewrite as files near the target size.
Cardinality estimate: The optimizer's guess of how many rows a step produces; the basis of memory grant sizes.Before running a query, the query optimizer guesses how many rows each step will produce. This guess is the cardinality estimate. Memory grants (Part 4) are built from it.
Clustering: Arranging rows physically so that units can be skipped for a given predicate.
Cold tier: Cheaper storage class with retrieval fees and minimum storage durations.
Commit: Finish a transaction successfully and make its changes permanent.
Commutative: Two operations give the same result in either order (for example two decrements).A combining operation is associative if grouping does not matter: (a + b) + c = a + (b + c). It is commutative if order does not matter: a + b = b + a. It is invertible if you can undo it: subtract to undo an addition. Sum and count have all three properties. Maximum is associative and commutative but not invertible. Median has none of the three in a useful form. These properties decide how cheaply an aggregate can be computed piece by piece.
Compaction: Rewriting many small files into fewer larger ones.An LSM-tree (Log-Structured Merge tree) is a storage structure that never changes files in place. It collects writes in memory, writes them as sorted, immutable files, and later compacts (merges and rewrites) the files in the background to remove old versions and deleted entries. A tombstone is a marker that says "this key was deleted"; compaction removes the key and the marker. Compaction costs disk and CPU. Write amplification is the ratio of bytes the storage actually writes to bytes the application asked to write. A Bloom filter is a small structure that answers "definitely not in this file" or "maybe in this file", so reads can skip most files.
Compare-and-swap (CAS): An atomic instruction that updates a variable only if it still has the expected old value.A lock or latch lets only one thread enter a piece of code or data at a time (a latch is a short lock inside a database). CAS (compare-and-swap) is one CPU instruction: "set this variable to new only if it still equals old". It either succeeds or fails and never blocks. Lock-free algorithms are built from it.
Copy-on-write: Writing a new copy of a unit and swapping a pointer instead of changing the old bytes.Two ways to handle a delete or update in immutable storage. Copy-on-write: rewrite the whole data file without the deleted rows. Merge-on-read: write a small side file (a delete vector or tombstone, a note that says "row 17 of file X is deleted") and let every reader apply it. Copy-on-write makes writes costly and reads clean. Merge-on-read makes writes cheap and every read does extra work.
Crypto-shredding: Deleting data by destroying its key.
Deletion vector: A compact bitmap of dead row positions, stored beside an immutable file.
Dormancy: The time an object goes without being read.
Entropy: The steady loss of good file size, order and metadata size as data arrives.1 other meaning →
Epoch-based reclamation (EBR): Freeing a retired node only after every thread has advanced past the epoch in which it was retired; cheap readers, unbounded memory if a thread stalls.
Fixed cost per operation (F): The cost paid for each request whatever its size.Most mechanisms charge a setup cost per operation, whatever its size. A flush to disk costs about the same for 1 byte or 100 KB. So doing many things in one operation (batching) is cheaper per item. The price of batching: the first item waits for the last.
Format migration: Moving data or metadata from one format version to another.
Garbage collection (GC): Runtime component that finds unreachable objects and reclaims their memory.In Java, Go, C# and similar languages you never call free. A background component, the garbage collector, finds objects that nothing refers to any more and reclaims their memory. A tracing collector does this by starting from known roots (variables) and following every reference to mark the objects that are still live. The live set is the total size of all live objects. A generational collector splits the heap into a young generation (new objects, collected often and cheaply) and an old generation (objects that survived several collections). Pauses are moments when the program is stopped so the collector can work.
Granularity: The smallest increment a mechanism handles or bills.The smallest piece a mechanism handles. If a courier charges per whole kilogram, one kilogram is the granularity, even if you ship 100 grams. When what you need is smaller than the granularity, you pay for the difference. This mechanism calls that difference granularity waste.
Growth factor (g): The ratio by which file size grows at each compaction step.
Hilbert / Morton (Z-order) curve: Space-filling curves that map several columns to one sort order.
Holistic statistic: A statistic (NDV, quantiles) that cannot be merged exactly from per-file values.
HyperLogLog: A distinct-count sketch that keeps the longest leading-zero run per bucket; error 1.04/√m.
Immutable: Never changed after it is written; it can be cached without invalidation.Something that is never changed after it is created. A new version is a new object with a new name. Parquet files in a data lake are usually immutable: you add files, you do not edit them.
Indirection: An extra lookup layer between a name and a physical location.A lookup layer between "the name you use" and "where the thing really is", like a phone book or a URL redirect. It lets you move things without telling everyone.
Legal hold: An outside order forbidding deletion of certain data.
LIST: An object-storage call that returns about 1,000 keys per request and must be called sequentially.
Liveness: Whether something may still be in use by a reader.1 other meaning →
Manifest: A small file that lists other files, with sizes, checksums and totals; it defines the set.A small file that lists other files (names, sizes, checksums). It says "this set of files is the dataset".
Merge-on-read: Writing small deltas and combining them with the base at read time.Two ways to handle a delete or update in immutable storage. Copy-on-write: rewrite the whole data file without the deleted rows. Merge-on-read: write a small side file (a delete vector or tombstone, a note that says "row 17 of file X is deleted") and let every reader apply it. Copy-on-write makes writes costly and reads clean. Merge-on-read makes writes cheap and every read does extra work.
NDV: Number of distinct values of a column; sizes hash aggregation memory.
Object storage: A cloud service holding named objects fetched by HTTP, replaced whole, never edited.A cloud service where each file ("object") has a name and is fetched with an HTTP request. A request costs 20-100 ms before the first byte arrives, and the provider charges a small fee per request. You cannot change a few bytes inside an object. You replace the whole object.
OCC (optimistic concurrency control): Run without locks, validate at commit, abort if something you read has changed.
OCC livelock: A long job that keeps losing commit races and never finishes.
Optimistic commit (CAS): Committing without locks: each writer prepares its change and swaps the table's current-version pointer with a compare-and-swap, which fails if another writer committed first; the loser checks for conflicts and retries.Writers do not lock the table. Each writer prepares its change, then tries to swap the "current version" pointer using compare-and-swap (CAS): "set the pointer to my new version, but only if it still points to the version I started from." If another writer committed first, the swap fails and the writer must check for conflict and retry. This is called optimistic concurrency control (OCC).
Orphan file: A file on storage that no metadata ever referenced.
Overlap depth: How many files' min/max ranges contain a random value; a clustering health metric.
Partition: A set of files proven to share a value; the claim that allows pruning.3 other meanings →A table in a data lake is really a set of files. A partition is a group of files that share a value, for example all files for date = 2025-03-01. Systems often store each partition in its own folder: orders/date=2025-03-01/part-0001.parquet.
Partition evolution: Changing a table's partition scheme over time.
Partition spec: The declared rule that divides a table into pieces by column value.
Pruning: Skipping whole partitions or files using metadata.A columnar file format (such as Parquet) stores each column of a table separately, so a query that needs 5 of 200 columns can read only those 5. Partitioning splits a table into folders (for example, one per day). Pruning (or skipping) means not reading the parts a query cannot need. Selectivity is the fraction of data a query actually needs.
Query planner: The part of a query engine that decides how to run a query (which files to read, in which order to join tables). It relies on statistics about the data, such as row counts.
Reachability: Whether any retained snapshot references a file.
Rebase: Re-applying a change on top of a newer table version.
Retention window: How long unreferenced files and old snapshots are kept; a bound on query duration.
Shuffle: Moving rows between stages so that data is partitioned by a different key.Two stages of a parallel query often need the data grouped by different keys. The shuffle moves every row from the machine that produced it to the machine that needs it. It is the most expensive data movement in a distributed engine.
Sketch (HyperLogLog): A small summary of a data set that answers one kind of question approximately.A quantile is a percentile: the value below which a given fraction of data falls (the median is the 50% quantile). A sketch is a small, fixed-size summary of a huge dataset that answers one kind of question approximately: for example a t-digest or KLL sketch answers quantile questions, HyperLogLog counts distinct values, and Space-Saving finds the most frequent values. A sketch is built in a single pass and uses bounded memory.
Snapshot: A frozen consistent view of the database at one moment.1 other meaning →A lock makes other transactions wait while you use a row. A snapshot is a frozen, consistent view of the database as it was at one moment; with it, readers do not need locks, because they read old versions while writers create new ones. This second idea is called MVCC (multi-version concurrency control): the database keeps several versions of each row. Both are mechanisms. This mechanism is about the promises they are used to keep.
Snapshot expiration: Removing old snapshots from metadata so their files can be reclaimed.
Square-root amortization: The best interval T = √(2C/r) when a fixed cost C fights a linearly growing cost r.
Table format: A specified metadata protocol that defines a table as a set of files plus commits.Object storage (like Amazon S3) stores whole files called objects behind HTTP requests. Parquet is a file format that stores data column by column. A table format (such as Iceberg or Delta) records, in small metadata files called manifests, which data files make up a table. A snapshot is one exact version of the table, identified by a snapshot id. A catalog is the service that tells you the current snapshot of a table.
Tiering: Moving data between storage classes according to expected access.1 other meaning →
Time travel: Reading a table as it was at an earlier moment (transaction time at storage level).
Tombstone: A marker record that says a key was deleted; must outlive every older copy.
Vacuum: PostgreSQL's collector for dead row versions.
Write amplification: Bytes written per byte of user data.If you write 1 MB of data and the system physically writes 10 MB (because of rewriting and merging), the write amplification is 10.
Write-Audit-Publish: Write to staging, check with assertions, then atomically publish.