Glossary

76 terms, in the sense this mechanism uses them. The platform glossary has every term across all mechanisms.

Absent / null / defaulted: Three different states of a field: not in the writer's schema, present with no value, present with a substituted default.
Backfill: Running a workflow for past intervals by iterating over the interval key space.1 other meaning →Going back over existing data to fill in or fix a value. Example: a new column exists, and a backfill job computes it for the last three years of rows.
Backward compatible: A new reader can read data written by an older writer.
Blast radius: The part of the system or the number of users affected when one failure happens.How much damage one mistake or one stolen secret can cause.
Broker: A service that holds records between producers and consumers.A broker (for example Kafka) is a server that stores a stream of messages. A topic is one named stream. Producers append to it. Consumers read from it, each at its own speed, sometimes days later.
CI (continuous integration): The automatic build and test run when code is proposed for merging.
CI schema diff: A check that compares the schema produced by code against the registered one and fails on breaking changes.
Closure property: A set of allowed changes that stays inside itself: applying changes in sequence never leaves the set.
Coercion: A read-time type conversion that may lose information; failure becomes null, error or truncation.
Column: One named field of a table.A table has rows (one record each) and columns (one named field each). A primary key is a column (or set of columns) that identifies one row. A foreign key (FK) is a column that holds the primary key of a row in another table. A join matches rows of two tables by key. Example: order_lines.product_id is a foreign key to products.product_id.
Column mapping: The rule that decides which bytes in a file belong to which schema field (by position, name or ID).
Consumer: A program that receives and processes records.A producer is a program that creates data and sends it somewhere. A consumer is a program that receives it and does something with it. A record (or message, or event) is one unit of data: for example, "order 4711 was placed". A broker is a middle service that holds records between producers and consumers (Kafka, RabbitMQ and Amazon SQS are brokers).
Consumer list: The registered list of readers of a table or topic; makes the contract phase of expand/contract decidable.
Contract (data): An agreement about meaning, owner, consumers, freshness and change process for a data product.
Contract phase: The last step of expand/contract: dropping the old field after proving nobody reads it.
DAG: Directed acyclic graph: tasks joined by arrows with no loops.Directed acyclic graph: boxes (tasks) joined by arrows (dependencies) with no loops. "Directed" means each arrow has a direction. "Acyclic" means you cannot follow the arrows and come back to where you started. An arrow A → B means "B may start after A finishes."
Dead-letter queue (DLQ): A holding area for records rejected by validation.A place where a pipeline puts records it could not process or did not accept, so they are kept instead of lost.
Default value: The value used when a field is missing.The value a system uses when a field is missing. Example: if priority is missing, use "STANDARD".
Deployment order: The order in which services are updated; usually not controlled.Services are updated one at a time, over minutes, days or never. At any moment, some run the old version and some the new one.
Dimension: A table of descriptive attributes of things (customer, product, date).In analysis, tables are often split into two kinds. A fact table records events: one row per sale, per click, per shipment. Its numbers are called measures (amount, quantity). A dimension table records descriptions of things: customers, products, dates, stores. Its columns are attributes (name, region, category). People filter and group by dimension attributes: "revenue by region for category = Tents".
Drift: Unplanned change in the shape or types of incoming data.
Dual-write: Writing both the old and the new field during expand/contract.
Enum: A type with a fixed set of allowed values.
Envelope versioning: Carrying a version field in the data and branching on it in the reader.
Expand/contract (parallel change): Add new, write both, backfill, migrate readers, verify, drop old. The safe way to rename.
Fail fast / fail early: Fail loudly at runtime as soon as a problem occurs / fail before anything runs (for example in CI).
Fail-closed / fail-open: On timeout, deny (closed) or proceed (open).
Field: One named value inside a record.A record (or row, or message) is one item of data, such as one event. A field is one named value inside it. The type says what kind of value is allowed.
Forward compatible: An old reader can read new data. Allows writers to be deployed first.
Full compatible: Both backward and forward compatible.
Idempotent: Doing it twice has the same effect as doing it once.An operation is idempotent if doing it twice has the same effect as doing it once. "Set the order status to SHIPPED" is idempotent. "Add 1 to the stock counter" is not. Idempotence is the main tool for living with duplicates.
Identity rule: The rule that defines "the same field": position, name or ID.
Immutable: Never changed after it is written; it can be cached without invalidation.Something that is never changed after it is created. A new version is a new object with a new name. Parquet files in a data lake are usually immutable: you add files, you do not edit them.
Inference: Reconstructing a schema from a sample of data.
Injective: A conversion where no two different inputs give the same output.
Lineage: The map of which assets feed which.A map of what feeds what: table A is read by job J, which writes table B. Following the map shows what would be affected if A changes.
Message topic: A broker (for example Kafka) is a server that stores a stream of messages. A topic is one named stream. Producers append to it. Consumers read from it, each at its own speed, sometimes days later.
Metadata: Data about data: files, schemas, owners, statistics, lineage.Data is the content: order rows, click events, images. Metadata is data about data: which files belong to a table, the column names, the owner, the number of rows, which job produced it. Metadata is usually tiny compared with the data, roughly one part in a thousand to one part in a million of the total size.
Metadata versus data: Data is the rows. Metadata is information about the rows (the schema, the file list). Changing metadata is fast and cheap. Rewriting data is slow and costly in proportion to its size.
Never reuse (ID rule): A retired column ID must never be assigned again.
Null: A present field that has no value.In SQL, NULL means "unknown". A condition can be TRUE, FALSE or UNKNOWN. NULL = NULL is UNKNOWN, not TRUE. A WHERE clause keeps only rows where the condition is TRUE. This small fact causes many of the corner cases here.
Nullable: A field is required if every record must have it. It is optional if a record may leave it out. A value can also be null, which means "present, but there is no value". The difference between "left out" and "null" will matter a lot in Part 4.
Optional: A field is required if every record must have it. It is optional if a record may leave it out. A value can also be null, which means "present, but there is no value". The difference between "left out" and "null" will matter a lot in Part 4.
Optional field: A field a record may leave out.
Parquet: Column-oriented file format with row groups, pages and a footer.Object storage (like Amazon S3) stores whole files called objects behind HTTP requests. Parquet is a file format that stores data column by column. A table format (such as Iceberg or Delta) records, in small metadata files called manifests, which data files make up a table. A snapshot is one exact version of the table, identified by a snapshot id. A catalog is the service that tells you the current snapshot of a table.
Partition spec: The declared rule that divides a table into pieces by column value.
Permissive parsing: Accepting records that do not fit by coercing, nulling or defaulting.
Producer: A program that creates and sends records.A producer is a program that creates data and sends it somewhere. A consumer is a program that receives it and does something with it. A record (or message, or event) is one unit of data: for example, "order 4711 was placed". A broker is a middle service that holds records between producers and consumers (Kafka, RabbitMQ and Amazon SQS are brokers).
Promotion (type): A total, value-preserving widening of a type.
Promotion path (variant): A rule for turning a frequently used variant path into a declared column.
Quarantine: Setting offending records aside and publishing the rest.
Rate threshold: A violation rate above which a pipeline escalates from per-record handling to stopping.
Reader: The writer is the program that creates the data under one schema. The reader is a program that reads it, with its own schema. A producer writes messages, a consumer reads them. We use both pairs of words as synonyms.
Reader-side default: A default applied when reading, kept in the reader's schema; changing it rewrites history silently.
Record: One unit of data sent from a producer to a consumer, such as "order 4711 was placed"; also called a message or an event. A broker holds records between the two.A producer is a program that creates data and sends it somewhere. A consumer is a program that receives it and does something with it. A record (or message, or event) is one unit of data: for example, "order 4711 was placed". A broker is a middle service that holds records between producers and consumers (Kafka, RabbitMQ and Amazon SQS are brokers).
Required: A field is required if every record must have it. It is optional if a record may leave it out. A value can also be null, which means "present, but there is no value". The difference between "left out" and "null" will matter a lot in Part 4.
Required field: A field every record must have.
Schema: The description of the shape of a record: fields, types, optionality.A description of the shape of a record: which fields (also called columns) exist, what type each has (integer, text, timestamp, ...), and which ones may be missing. Example: a shipment event has shipment_id (text), status (text), weight (number), occurred_at (timestamp).
Schema registry: A service that stores schemas and returns an ID for each.A small central service that stores every version of every schema and checks new versions against rules. Producers and consumers ask it which schema applies. We study it in Part 5.
Schema-on-read / schema-on-write: Structure is checked when reading, or when writing.
Semantic type: A type that carries meaning or unit (for example money in cents).
Serialization: Turning in-memory values into a flat sequence of bytes.Programs hold values in memory (objects, structs, lists). To send them over a network or store them in a file, you must turn them into a flat sequence of bytes. This is serialization; the reverse is deserialization (or decoding). JSON is a serialization format. So are Protocol Buffers (Protobuf), Avro and Thrift.
Shredding: Projecting self-describing documents into typed columns.
Stop-the-line: Halting a pipeline when data violates the rules.
Subject (registry): The name under which a series of schema versions is stored.
Subject strategy: The choice of contract granularity: per topic, per record type, or both.
Table format: A specified metadata protocol that defines a table as a set of files plus commits.Object storage (like Amazon S3) stores whole files called objects behind HTTP requests. Parquet is a file format that stores data column by column. A table format (such as Iceberg or Delta) records, in small metadata files called manifests, which data files make up a table. A snapshot is one exact version of the table, identified by a snapshot id. A catalog is the service that tells you the current snapshot of a table.
Topic: A named stream of records.A topic is a named stream of records ("orders"). Big topics are cut into partitions: separate ordered lists that can be stored and read in parallel. Records in one partition have a fixed order. Across partitions there is no shared order.
Total (function): A conversion that has an answer for every input.
Transaction: A group of operations executed as one all-or-nothing unit.A group of reads and writes that the database treats as one unit: either all of it takes effect, or none of it does. In SQL you mark it with BEGIN ... COMMIT (make it permanent) or ROLLBACK (throw it away). Example: moving 30 from Anna's wallet to Ben's wallet.
Transitive compatibility: Checking a new schema against every earlier version, not only the last.
Type: A record (or row, or message) is one item of data, such as one event. A field is one named value inside it. The type says what kind of value is allowed.
Union type: A field that may be one of an enumerated set of types.
Variant: A column whose value is an arbitrary self-describing structure.
Writer: The writer is the program that creates the data under one schema. The reader is a program that reads it, with its own schema. A producer writes messages, a consumer reads them. We use both pairs of words as synonyms.
Writer-side default: A default written into the data at write time; it is frozen.