Glossary

70 terms, in the sense this mechanism uses them. The platform glossary has every term across all mechanisms.

Aliasing: Losing intermediate values because you observe a changing thing less often than it changes.If you look at something only every 15 minutes, you see its state at those moments and nothing in between. If a value changes three times between two looks, you see only the last value. Signal engineers call this aliasing.
Analytics: Large scans and aggregations over historical data, run on a warehouse or lake kept separate from the application's OLTP database (the one handling many small reads and writes).An OLTP database (online transaction processing) is the database behind an application. It handles many small reads and writes. An analytics warehouse (or data lake) is a separate system built for large scans and aggregations. Copying data from the first to the second is the job of this mechanism.
Anti-entropy: A background process that compares replicas and repairs differences; the only repair method that guarantees convergence.
Backfill: Running a workflow for past intervals by iterating over the interval key space.1 other meaning →Going back over existing data to fill in or fix a value. Example: a new column exists, and a backfill job computes it for the last three years of rows.
Before-image: The old version of a row, included in a change event; allows an update to be retracted.
Blue-green (table level): Building a new table in parallel with the old one, diffing, then switching by a pointer swap.
Change data capture (CDC): Turning a database's changes into a stream of records.A feed of every change (insert, update, delete) made to a database table, delivered as a stream of events.
Commit: Finish a transaction successfully and make its changes permanent.
Completeness gap: The set of writes that a detection mechanism cannot see.
Copy-on-write: Writing a new copy of a unit and swapping a pointer instead of changing the old bytes.Two ways to handle a delete or update in immutable storage. Copy-on-write: rewrite the whole data file without the deleted rows. Merge-on-read: write a small side file (a delete vector or tombstone, a note that says "row 17 of file X is deleted") and let every reader apply it. Copy-on-write makes writes costly and reads clean. Merge-on-read makes writes cheap and every read does extra work.
Cursor: A consumer's remembered position in a change stream (offset, LSN or snapshot id).1 other meaning →A bookmark that says "I have read everything up to here". It can be a byte offset in a file, a row number, a timestamp or an opaque token.
DDL: Statements that change a table's shape (ALTER, DROP, TRUNCATE).DML (data manipulation language) is INSERT, UPDATE, DELETE. DDL (data definition language) changes the shape of the table: ALTER TABLE, DROP COLUMN, TRUNCATE.
Deletion vector: A compact bitmap of dead row positions, stored beside an immutable file.
Differential dataflow: A research approach that maintains differences between versions, making recursive computations incrementally maintainable.
DML: Statements that change rows (INSERT, UPDATE, DELETE).DML (data manipulation language) is INSERT, UPDATE, DELETE. DDL (data definition language) changes the shape of the table: ALTER TABLE, DROP COLUMN, TRUNCATE.
Dual-write: Writing both the old and the new field during expand/contract.
Envelope: The structure around a change event: op, images, position, transaction id, timestamps.
Event stream: An ordered, append-only sequence of events, stored by a broker such as Kafka in topics split into partitions. Order holds within a partition, and each reader tracks its position as an offset.A message broker (Kafka is the best-known) stores events in append-only topics. A topic is split into partitions to allow parallel reading. Events inside one partition keep their order; events in different partitions have no order between them. A reader remembers its position in each partition as an offset (a "cursor").
Exactly-once: Each message's effect happens once; real only within one system's transaction boundary.A processing guarantee that each input affects the result exactly one time, even when machines crash and retry. It is hard; we will only need the two basic ideas (idempotence, and committing a position together with the state) when it appears.
Fidelity ladder: The ordering of change-detection methods from most complete and coupled to cheapest and loosest.
Fixed overhead (F): The cost of an incremental pipeline that does not shrink with the amount of change.
Fold: Applying events in order to produce a state.
Freshness floor: The minimum delay between a source change and its visibility, set by the commit cadence.
Group (algebra): A monoid in which every change can be undone (an inverse exists); needed to support deletions incrementally.
Hard delete: Physically removing a row, leaving no row to find.
Idempotent: Doing it twice has the same effect as doing it once.An operation is idempotent if doing it twice has the same effect as doing it once. "Set the order status to SHIPPED" is idempotent. "Add 1 to the stock counter" is not. Idempotence is the main tool for living with duplicates.
Immutable: Never changed after it is written; it can be cached without invalidation.Something that is never changed after it is created. A new version is a new object with a new name. Parquet files in a data lake are usually immutable: you add files, you do not edit them.
Incremental snapshot: Bootstrapping a table in key-range chunks with watermark markers in the log, without long locks.
Incremental view maintenance (IVM): Updating a saved query result from the change to its input, without recomputing it.
Kappa-style reprocessing: Fixing code and replaying the retained stream to rebuild results.
Logical decoding: Translating the physical log into row-level change records.
LSN / SCN / GTID: A number identifying a position in the source's log or transaction order.
Materialized view: A stored query result that the system refreshes, with its admission declared.A query result that is saved as a table and kept up to date, instead of being recomputed each time it is read.
MERGE: A SQL statement that updates matching rows and inserts the rest.An upsert means "update the row if the key exists, insert it if it does not." MERGE is the SQL statement that does this (and can also delete).
MERGE / upsert: Update the row if the key matches, insert if not (and optionally delete).
Merge penalty (c′/c): How much more a random-access merge costs per row than a sequential scan (often 10–100×).
Merge-on-read: Writing small deltas and combining them with the base at read time.Two ways to handle a delete or update in immutable storage. Copy-on-write: rewrite the whole data file without the deleted rows. Merge-on-read: write a small side file (a delete vector or tombstone, a note that says "row 17 of file X is deleted") and let every reader apply it. Copy-on-write makes writes costly and reads clean. Merge-on-read makes writes cheap and every read does extra work.
Multiset: A collection of values in which duplicates are allowed and counted.
Offset: A consumer's position in a partition (one integer).An event is a small record: "order 77 placed by user 12 at 12:03:04". A log is an append-only file or service: new events are added at the end, and old events are never changed. A topic is a named log (the word comes from systems like Apache Kafka). Every event has an offset, its position in the log (0, 1, 2, …). A reader remembers "I have read up to offset 5,000", and can go back to offset 3,000 and read again. This ability to replay is the foundation of this whole mechanism. A log keeps events for a configured time (retention), for example 7 days, then deletes the oldest ones.
OLTP: Online transaction processing: many small reads and writes.An OLTP database (online transaction processing) is the database behind an application. It handles many small reads and writes. An analytics warehouse (or data lake) is a separate system built for large scans and aggregations. Copying data from the first to the second is the job of this mechanism.
Outbox (transactional): Table written in the same transaction as the state change, relayed to a broker later.
Outbox tailing: A separate process reading new outbox rows and publishing them.
Partition: One ordered slice of a topic that can be stored and read in parallel.3 other meanings →A topic is a named stream of records ("orders"). Big topics are cut into partitions: separate ordered lists that can be stored and read in parallel. Records in one partition have a fixed order. Across partitions there is no shared order.
Path dependence: Correctness that depends on the entire history of runs, not only on the current input.
Primary key: A unique, non-null identifier for each row.Rules a database can enforce. A primary key says each row has a unique, non-null identifier. A unique constraint says no two rows share a value. A foreign key says a value must exist in another table (for example every orders.customer_id must exist in customers). A check constraint says a row must satisfy a predicate.
Query/diff-based CDC: Detecting changes by polling a query, such as updated_at > watermark.
Re-processing window: Always re-running the last N periods idempotently to catch late data.
Replay: Re-reading records that were already processed.
Retraction: Removing the effect of an earlier input, expressed as a −1 change.
Row: A table has rows (one record each, such as one scan event) and columns (one attribute each, such as weight_g). A table with n columns has n values in each row.
Sampling: Recording the state periodically; cannot see events shorter than the interval.1 other meaning →If you look at something only every 15 minutes, you see its state at those moments and nothing in between. If a value changes three times between two looks, you see only the last value. Signal engineers call this aliasing.
SCD (slowly changing dimension): A dimension whose attributes change over time; handled by Types 1, 2, 3, 4 and 6.
Schema registry: A service that stores schemas and returns an ID for each.A small central service that stores every version of every schema and checks new versions against rules. Producers and consumers ask it which schema applies. We study it in Part 5.
Semigroup: A set with an operation but no undo; MAX is the example.
Shadow table: A table that stores change records (trigger CDC) or a new version of a table under construction (blue-green).
Signed multiplicity: Representing changes as rows with counts: insert +1, delete −1, update = −old and +new.
Sketch (HyperLogLog): A small summary of a data set that answers one kind of question approximately.A quantile is a percentile: the value below which a given fraction of data falls (the median is the 50% quantile). A sketch is a small, fixed-size summary of a huge dataset that answers one kind of question approximately: for example a t-digest or KLL sketch answers quantile questions, HyperLogLog counts distinct values, and Space-Saving finds the most frequent values. A sketch is built in a single pass and uses bounded memory.
Snapshot: A frozen consistent view of the database at one moment.1 other meaning →A lock makes other transactions wait while you use a row. A snapshot is a frozen, consistent view of the database as it was at one moment; with it, readers do not need locks, because they read old versions while writers create new ones. This second idea is called MVCC (multi-version concurrency control): the database keeps several versions of each row. Both are mechanisms. This mechanism is about the promises they are used to keep.
Snapshot-diff: Finding changes by comparing the file lists of two snapshots.
State-convergent: A consumer for which only the final state matters, so replays and aliasing are harmless.
Table: Rows of records with named columns. Keys identify rows, and foreign keys link a row to a row in another table.A table has rows (one record each) and columns (one named field each). A primary key is a column (or set of columns) that identifies one row. A foreign key (FK) is a column that holds the primary key of a row in another table. A join matches rows of two tables by key. Example: order_lines.product_id is a foreign key to products.product_id.
Tail (a log): The slowest requests, such as those above p99.p50 is the median latency, p99 is the latency that 99% of requests beat. The slowest requests are the tail. When one request needs answers from many servers (a fan-out) it must wait for the slowest one. With 100 servers that each have a 1% chance of being slow, the chance that at least one is slow is 1 − 0.99¹⁰⁰ = 63%. So the user-facing tail is much worse than any single server's tail.
Topic: A named stream of records.A topic is a named stream of records ("orders"). Big topics are cut into partitions: separate ordered lists that can be stored and read in parallel. Records in one partition have a fixed order. Across partitions there is no shared order.
Transaction: A group of operations executed as one all-or-nothing unit.A group of reads and writes that the database treats as one unit: either all of it takes effect, or none of it does. In SQL you mark it with BEGIN ... COMMIT (make it permanent) or ROLLBACK (throw it away). Example: moving 30 from Anna's wallet to Ben's wallet.
Trigger: A database procedure that runs automatically on each row change.1 other meaning →
Two-phase commit (2PC): Prepare then decide; participants give up the right to abort after voting yes.A way to make a change appear in two places atomically (all or nothing). Phase 1: every participant prepares and promises it can commit. Phase 2: when all have promised, everyone commits. If any participant says no, everyone aborts. Until phase 2, the change stays invisible.
Type 2 dimension: An SCD that adds a new row for each change.
Upsert: Insert-or-replace by key.A primary key is a column (or set of columns) whose value is unique per row. An upsert means "insert this row, or update it if the key already exists". In SQL: INSERT ... ON CONFLICT (order_id) DO UPDATE ....
Watermark: The highest timestamp, partition or position already processed.1 other meaning →
Write-ahead log (WAL): Write-ahead log: ordered, append-only description of changes, forced before commit.Before a database changes its data pages, it first appends a description of the change to a sequential file called the log. If the machine crashes, the database replays the log to recover. Every serious database has one (it may be called WAL, redo log or binlog). Each position in the log has an increasing number, called an LSN (log sequence number; other systems call it SCN or GTID). In this mechanism the key fact is: the log is a complete, ordered record of every change, and it already exists..