Glossary

80 terms, in the sense this mechanism uses them. The platform glossary has every term across all mechanisms.

Advisory metadata: Metadata that informs people but is not needed to read the data (lineage, owner, glossary). Errors are silent.
Asset identifier (asset ID): A stable, meaningless identifier for a table or other asset, so that names can change without breaking links.
Atomic: A change that is either fully visible or not visible at all.A change that is either fully visible or not visible at all. Renaming a single file on one filesystem is usually atomic: other programs see the old name or the new name, never half of a rename.
Audit trail: A complete, tamper-evident, long-retained record of actions, used as evidence.
Big-O notation: O(files) means "the work grows in proportion to the number of files." O(1) means "the work is the same however many files exist."
BM25: A standard text relevance score that sums per-term contributions based on term frequency, document length and rarity.
Branch: A movable named pointer over shared immutable history, allowing changes to be isolated.
Business glossary: A set of human-written definitions of business terms and their links to data columns. Asserted metadata.
Cache: A deliberately kept second copy of data that no protocol forces to stay correct.1 other meaning →A cache is a small, fast place that holds copies of data kept somewhere bigger and slower. When a request finds its data in the cache, that is a hit. When it does not, it is a miss, and the system must go to the slower place (the backing store or origin). The hit rate h is the fraction of requests that hit. The miss penalty is the extra time a miss costs.
Cardinality: The number of distinct values a label can take. The series count is the product over all labels.2 other meanings →The number of distinct things in a set. "20 million files" is a statement about cardinality. Metadata size follows cardinality (how many objects), not bytes.
Catalog: The service that knows which tables exist, where they are and their definitions.A catalog is the service that knows which tables exist and where they are. A metastore is the database inside (or behind) a catalog that stores table definitions, partitions and file lists. Compare it with a library card index: it tells you where each book sits and which books exist, without opening any book.
Column-level lineage: Lineage that tracks how each output column is derived from input columns.
Compare-and-swap (CAS): An atomic instruction that updates a variable only if it still has the expected old value.A lock or latch lets only one thread enter a piece of code or data at a time (a latch is a short lock inside a database). CAS (compare-and-swap) is one CPU instruction: "set this variable to new only if it still equals old". It either succeeds or fails and never blocks. Lock-free algorithms are built from it.
Coverage (lineage): The fraction of computation that emits lineage metadata at all.
Crawling: Pulling metadata by periodically walking systems from outside.
Crypto-shredding: Deleting data by destroying its key.
Dangling reference: Metadata that points at a file that does not exist; queries fail.
Data: The content a system stores and moves (order rows, click events, images), as opposed to metadata, the much smaller description of it: files, columns, owner, row counts, lineage.Data is the content: order rows, click events, images. Metadata is data about data: which files belong to a table, the column names, the owner, the number of rows, which job produced it. Metadata is usually tiny compared with the data, roughly one part in a thousand to one part in a million of the total size.
Default-deny: A policy that refuses access unless access is explicitly granted; unlabelled data is refused.
Defence in depth: Using several independent controls so that one failure does not defeat the protection.
Deletion vector: A compact bitmap of dead row positions, stored beside an immutable file.
Dictionary encoding: Replacing each value by a small integer index into a table of distinct values.
Fail closed: When information is missing, refuse rather than allow.
Fast-forward merge: A merge in which one branch has not moved since the split, so the other is simply adopted.
File: A table in a data lake is really a set of files. A partition is a group of files that share a value, for example all files for date = 2025-03-01. Systems often store each partition in its own folder: orders/date=2025-03-01/part-0001.parquet.
Graph: A graph is nodes joined by edges (here: tables joined by "feeds into"). The transitive closure of a node is everything reachable from it by following edges any number of steps: "everything downstream of A."
Hash-chained log: A log in which each entry contains the hash of the previous one, so edits can be detected.
HyperLogLog: A distinct-count sketch that keeps the longest leading-zero run per bucket; error 1.04/√m.
Idempotent / safe to repeat: An operation that gives the same result if executed more than once.
Immutable: Never changed after it is written; it can be cached without invalidation.Something that is never changed after it is created. A new version is a new object with a new name. Parquet files in a data lake are usually immutable: you add files, you do not edit them.
Indirection: An extra lookup layer between a name and a physical location.A lookup layer between "the name you use" and "where the thing really is", like a phone book or a URL redirect. It lets you move things without telling everyone.
Invalidation: A cache is a saved copy of a fact kept somewhere closer or cheaper to read. Invalidation is the signal "the original changed, throw away your copy." A copy without such a signal slowly becomes wrong.
Inverted index: A map from each term to the list of documents that contain it.
Late binding: Checking the format only when data is read, so errors appear in the consumer, far from the cause.Binding means fixing the structure and meaning of data. Early binding: decide at load time ("this column is the customer's region"). Late binding: store raw facts and decide at read time. Early binding makes reads simple and changes expensive. Late binding makes changes cheap and reads complex.
Legal hold: An outside order forbidding deletion of certain data.
Lineage: The map of which assets feed which.A map of what feeds what: table A is read by job J, which writes table B. Following the map shows what would be affected if A changes.
Linearization point: The instant at which an operation takes effect for all observers.
Load-bearing metadata: Metadata the data cannot be read without; errors surface at once.
Mandatory channel: A path all use must pass through (for example deny-and-log), turning absence of evidence into enforced absence.
Manifest: A small file that lists other files, with sizes, checksums and totals; it defines the set.A small file that lists other files (names, sizes, checksums). It says "this set of files is the dataset".
Manifest list / snapshot: A file that lists the manifests that make up one version of a table.
Mergeable statistic: A statistic for two data sets that can be computed from the statistic of each, without re-reading the data.
Metadata: Data about data: files, schemas, owners, statistics, lineage.Data is the content: order rows, click events, images. Metadata is data about data: which files belong to a table, the column names, the owner, the number of rows, which job produced it. Metadata is usually tiny compared with the data, roughly one part in a thousand to one part in a million of the total size.
Metastore: The database holding table definitions, partitions and file lists.A catalog is the service that knows which tables exist and where they are. A metastore is the database inside (or behind) a catalog that stores table definitions, partitions and file lists. Compare it with a library card index: it tells you where each book sits and which books exist, without opening any book.
Micro-batching: Time-based batching with a very short interval (about 100 ms to seconds).
Min/max statistics: The smallest and largest value of a column in a block, used to decide skipping.
Object storage: A cloud service holding named objects fetched by HTTP, replaced whole, never edited.A cloud service where each file ("object") has a name and is fetched with an HTTP request. A request costs 20-100 ms before the first byte arrives, and the provider charges a small fee per request. You cannot change a few bytes inside an object. You replace the whole object.
OCC (optimistic concurrency control): Run without locks, validate at commit, abort if something you read has changed.
OLTP: Online transaction processing: many small reads and writes.An OLTP database (online transaction processing) is the database behind an application. It handles many small reads and writes. An analytics warehouse (or data lake) is a separate system built for large scans and aggregations. Copying data from the first to the second is the job of this mechanism.
Order of magnitude: 10³ = 1,000 and 10⁻⁶ = one millionth. "Two to three orders of magnitude" means 100 to 1,000 times.
Orphan file: A file on storage that no metadata ever referenced.
PageRank: An algorithm that ranks items by how many important items depend on them.
Partition: A set of files proven to share a value; the claim that allows pruning.3 other meanings →A table in a data lake is really a set of files. A partition is a group of files that share a value, for example all files for date = 2025-03-01. Systems often store each partition in its own folder: orders/date=2025-03-01/part-0001.parquet.
Partition pruning: Skipping partitions that cannot match the query's filter.A zone map stores the minimum and maximum of a column for each block of rows. If your range does not overlap a block's min/max, you skip the block without reading it. Partition pruning is the same idea at table level: a table is split into partitions by a key (for example by date), and the engine skips partitions that cannot match. Both rely on values being comparable, so that "min" and "max" mean something.
Planner (optimizer): The component that chooses how a query is executed.The query planner decides how to run a query: scan the whole table, or use an index, and which one. It guesses how many rows each step returns using statistics about the data. EXPLAIN shows the chosen plan. An index that exists but is not chosen costs you everything and buys you nothing.
Pointer: A pointer is an address stored inside a data item that says where the next item is. A dependent fetch (or dependent miss) is a memory or disk read whose address you only learn from the previous read. Following a linked list is the typical example: you cannot read item 5 until you have read item 4. Dependent fetches cannot overlap, so each one pays the full latency. A cache miss to RAM costs about 80 ns. An NVMe SSD read costs about 100 µs (1 µs = 1/1,000,000 s; 1 ns = 1/1,000,000,000 s). A read from cloud object storage costs 20 to 100 ms.
Propagation (of labels): Automatically passing a label from a source column to everything derived from it.
Provenance: A per-row record of origin: source, time and pipeline version.
Reconciliation: Comparing data to a second independent source.Checking data against a second, independent source ("our revenue total equals the payment provider's total").
Run-length encoding (RLE): Storing (value, count) pairs for runs of equal values.
Schema registry: A service that stores schemas and returns an ID for each.A small central service that stores every version of every schema and checks new versions against rules. Producers and consumers ask it which schema applies. We study it in Part 5.
Semantic layer: A central definition of metrics and join paths, enforced by a query compiler.
Sensitivity label: A tag such as "personal data" attached to a column.
Sketch (HyperLogLog): A small summary of a data set that answers one kind of question approximately.A quantile is a percentile: the value below which a given fraction of data falls (the median is the 50% quantile). A sketch is a small, fixed-size summary of a huge dataset that answers one kind of question approximately: for example a t-digest or KLL sketch answers quantile questions, HyperLogLog counts distinct values, and Space-Saving finds the most frequent values. A sketch is built in a single pass and uses bounded memory.
Snapshot: A frozen consistent view of the database at one moment.1 other meaning →A lock makes other transactions wait while you use a row. A snapshot is a frozen, consistent view of the database as it was at one moment; with it, readers do not need locks, because they read old versions while writers create new ones. This second idea is called MVCC (multi-version concurrency control): the database keeps several versions of each row. Both are mechanisms. This mechanism is about the promises they are used to keep.
Sound but not complete: Everything reported is true, but not everything true is reported.
Static parsing: Reading code without running it to find which assets it uses.
Statistics: Per-column summaries (NDV, histogram, most common values) used for estimates.The database cannot look at all data for every query, so it keeps a small summary per column, built by a command (ANALYZE) that reads a sample. The summary includes the number of distinct values (NDV), the most common values, and a histogram (a list of ranges, called buckets, with roughly how many values fall in each). The optimizer uses these statistics to produce its estimates. The collection of tables, columns, constraints and statistics that the database knows about is its catalog (or system catalog).
Stewardship: The responsibility of caring for the quality and meaning of a data asset.
Surrogate key: A meaningless warehouse-assigned key that identifies a row (a version).A natural key (or business key) is an identifier from the real world or the source system: an email, an order number, a VAT number. A surrogate key is a meaningless number that the warehouse invents, such as 10482.
Table: Rows of records with named columns. Keys identify rows, and foreign keys link a row to a row in another table.A table has rows (one record each) and columns (one named field each). A primary key is a column (or set of columns) that identifies one row. A foreign key (FK) is a column that holds the primary key of a row in another table. A join matches rows of two tables by key. Example: order_lines.product_id is a foreign key to products.product_id.
Table-level lineage: Lineage between whole tables.
Time travel: Reading a table as it was at an earlier moment (transaction time at storage level).
Transactional outbox: A pattern in which a message to send is stored in the same transaction as the change, and sent later.
Transitive closure: Everything reachable from a node by following edges any number of steps.A graph is nodes joined by edges (here: tables joined by "feeds into"). The transitive closure of a node is everything reachable from it by following edges any number of steps: "everything downstream of A."
TTL: Time to live: how long an entry may be served before it expires.A lifetime attached to a cache entry. After the TTL expires, the entry is dropped or treated as missing.
Vacuum: PostgreSQL's collector for dead row versions.
Write-ahead log (WAL): Write-ahead log: ordered, append-only description of changes, forced before commit.Before a database changes its data pages, it first appends a description of the change to a sequential file called the log. If the machine crashes, the database replays the log to recover. Every serious database has one (it may be called WAL, redo log or binlog). Each position in the log has an increasing number, called an LSN (log sequence number; other systems call it SCN or GTID). In this mechanism the key fact is: the log is a complete, ordered record of every change, and it already exists..
Write-Audit-Publish: Write to staging, check with assertions, then atomically publish.
Zone map: Per-file min and max values of a column, used to skip files.A zone map stores the minimum and maximum of a column for each block of rows. If your range does not overlap a block's min/max, you skip the block without reading it. Partition pruning is the same idea at table level: a table is split into partitions by a key (for example by date), and the engine skips partitions that cannot match. Both rely on values being comparable, so that "min" and "max" mean something.