Glossary

95 terms, in the sense this mechanism uses them. The platform glossary has every term across all mechanisms.

Adoption (as a resource): An interface wins because many tools already use it; the same installed base later makes it hard to change.
Allow-listed queries: A GraphQL defence: only pre-approved query shapes are accepted, which re-creates fixed endpoints.
Arrow / Arrow Flight: Arrow is a columnar in-memory layout; Flight is an RPC system that sends Arrow data directly over the network.
Atomic: A change that is either fully visible or not visible at all.A change that is either fully visible or not visible at all. Renaming a single file on one filesystem is usually atomic: other programs see the old name or the new name, never half of a rename.
Authentication token expiry: A credential that stops working at a fixed time, so failure is a cliff, even in the middle of a long transfer.
Avro: Row serialization format with zero per-field framing; the reader needs the writer's schema.
Back-channel: A way for the receiver to answer the sender (errors, limits, questions). Files have none.A way for the receiver to answer the sender. A file on disk has no back-channel: you cannot ask a file what it meant. A web request has one: the server can reply with an error, a limit or a question.
Back-pressure: A slow receiver slowing down the sender through flow control.
Backfill: Running a workflow for past intervals by iterating over the interval key space.1 other meaning →Going back over existing data to fill in or fix a value. Example: a new column exists, and a backfill job computes it for the last three years of rows.
Backoff (exponential): Waiting longer after each failed attempt, usually doubling.
Binding (early/late): Fixing a decision; early binding is before seeing the data, late binding is after.
Boundary: The place where one party's control ends and another's begins; no change is atomic there.1 other meaning →The place where one party's control ends and another's begins. Inside your own system you can change the writer and the reader in one deployment. At a boundary you cannot.
Buffering: Storing excess work inside the system; latency and memory grow without bound.Latency is how long one operation takes. Throughput is how much work finishes per second. Buffering means putting a queue or a temporary store between a fast side and a slow side, so neither one waits for the other.
Cache line: The 64-byte block the CPU moves between RAM and cache.Memory (RAM) is fast but small and loses its content when power is off. Storage (SSD, disk) is slower but large and permanent. Hardware never moves single bytes. It moves fixed-size blocks:
Catalog: The service that knows which tables exist, where they are and their definitions.A catalog is the service that knows which tables exist and where they are. A metastore is the database inside (or behind) a catalog that stores table definitions, partitions and file lists. Compare it with a library card index: it tells you where each book sits and which books exist, without opening any book.
Catalog API: A catalog offered as a service that engines call; it commits metadata and can grant scoped credentials.
Change data capture (CDC): Turning a database's changes into a stream of records.A feed of every change (insert, update, delete) made to a database table, delivered as a stream of events.
Checksum: A small value computed from data, used to detect changes or corruption.
Collation: The rules for comparing and sorting text; different rules give different orders.
Column-oriented: Storing each column's values together instead of each row together. Good for queries that read a few columns, and for compression, because one column holds similar values.Row layout stores each record together: [id, name, price], then the next record. Column layout stores all id values together, then all name values, then all price values. Column layout is good when a query needs only a few columns. It is also good for compression, because one column holds similar values.
Columnar format: A layout storing all values of one column together (Parquet, ORC).
Complexity budget: A GraphQL defence that rejects queries whose estimated cost is too high.
Connector framework: A source-transport-sink structure that integrates each system once and manages the bookmark.
Consumer: A program that receives and processes records.A producer is a program that creates data and sends it somewhere. A consumer is a program that receives it and does something with it. A record (or message, or event) is one unit of data: for example, "order 4711 was placed". A broker is a middle service that holds records between producers and consumers (Kafka, RabbitMQ and Amazon SQS are brokers).
Content negotiation: Both parties announce what they support and settle on the intersection at run time.
Control total: A count or sum (rows, amount) in a manifest used to check that a batch is complete.
Cost of change: Dual-running time × number of counterparties.
Cost relocation: Costs are not removed by a design, only moved to another party, time or system.Across the platform, costs are never removed. A design that seems to remove a cost has moved it: to the writer, to the reader, to another team, to a new dependency. The useful question is always "who pays now, and can they refuse?"
Cursor: A consumer's remembered position in a change stream (offset, LSN or snapshot id).1 other meaning →A bookmark that says "I have read everything up to here". It can be a byte offset in a file, a row number, a timestamp or an opaque token.
Dead-letter path: A place where messages go after too many failed delivery attempts.
Declarative: Describing what result you want (SQL) and letting the callee decide how.
Deprecation / sunset: The planned retirement of an old interface version after a dual-running period.
Dialect: A vendor's variant of SQL, caused by differences in capability.
Drift: Unplanned change in the shape or types of incoming data.
Dual-running: The period in which old and new contracts both work.
Duplicate: If a request gets no answer, the sender sends it again (a retry). If the first one actually worked, the receiver now sees the same request twice (a duplicate).
ELT: Extract, Load, Transform: land raw data first, transform in the destination.
ETL: Extract, Transform, Load: transform before the destination receives data.
External table: A table registered over files you do not own or ingest.
Federation: One engine querying data in another system without copying it first.1 other meaning →
FlatBuffers: A format whose bytes can be used directly as in-memory values.
Footer: Metadata at the end of a file, such as a directory of offsets and statistics.
Framing: The method that lets a receiver find message boundaries inside a TCP byte stream.
GraphQL: An API style in which the caller specifies the shape of the response.
gRPC: An RPC framework with a definition file that generates code for both sides and supports streaming.
Idempotency key: Unique identifier of an intent, used to make retries safe.
Idempotent: Doing it twice has the same effect as doing it once.An operation is idempotent if doing it twice has the same effect as doing it once. "Set the order status to SHIPPED" is idempotent. "Add 1 to the stock counter" is not. Idempotence is the main tool for living with duplicates.
Inference: Reconstructing a schema from a sample of data.
Interop hazard surface: Agreement-level differences (encoding, decimals, nulls, time, integer width, collation) that fail silently.
Ipc: Inter-process communication: moving data between programs.1 other meaning →
JDBC / ODBC: Standard driver interfaces for SQL databases, built around a row cursor.
Keyset pagination: Paging by "everything after this key" instead of by count.
Latency: Time one operation takes from request to response.Throughput is work finished per second. Latency is how long one operation takes. Little's Law links them to the number of operations in progress: L = λ × W, where λ is the arrival rate (per second), W is the average time one operation spends in the system, and L is how many are inside the system at the same time. Example: 1,000 requests per second, each taking 20 ms (0.02 s), means 1,000 × 0.02 = 20 requests in flight at any moment. This is an identity, not a design choice.
Length-prefixed: A value stored as its length followed by its bytes.
Manifest: A small file that lists other files, with sizes, checksums and totals; it defines the set.A small file that lists other files (names, sizes, checksums). It says "this set of files is the dataset".
MessagePack: A compact binary encoding that resembles JSON.
Monotonic: Only increases; never goes backward.
N × M / N + M: Connections needed without a shared standard versus with one.
N-by-M problem: If N tools must each talk to M systems, you need N × M separate connections. If every tool and every system speaks one shared standard, you need only N + M adapters.
Narrow waist: A single thin interface through which everything passes.
NDJSON: Newline-delimited JSON: one record per line.
Null / empty / absent: Three different states (unknown, empty string, missing field) that formats often confuse.
Offset: A consumer's position in a partition (one integer).An event is a small record: "order 77 placed by user 12 at 12:03:04". A log is an append-only file or service: new events are added at the end, and old events are never changed. A topic is a named log (the word comes from systems like Apache Kafka). Every event has an offset, its position in the log (0, 1, 2, …). A reader remembers "I have read up to offset 5,000", and can go back to offset 3,000 and read again. This ability to replay is the foundation of this whole mechanism. A log keeps events for a configured time (retention), for example 7 days, then deletes the oldest ones.
Offset pagination: Paging by "skip N, take M"; wrong under concurrent writes and slow for large N.
Opaque token: A position whose meaning only the provider knows.
ORC / Parquet: Columnar file formats with footers and statistics.
Parse tax: The work the reader does turning bytes into values, on every read.
Parsing: Serializing means turning values in memory into bytes (to store or send). Parsing (or deserializing) is the reverse: turning bytes into values. The work of parsing is the parse tax here: a cost paid by the reader every time it reads.
Partial success: A batch where some items succeeded and some failed; must be reported explicitly.
Position: A bookmark saying how far a consumer has read.A bookmark that says "I have read everything up to here". It can be a byte offset in a file, a row number, a timestamp or an opaque token.
Postgres wire protocol: The PostgreSQL client-server protocol, copied by other systems to inherit its tools.
Producer: A program that creates and sends records.A producer is a program that creates data and sends it somewhere. A consumer is a program that receives it and does something with it. A record (or message, or event) is one unit of data: for example, "order 4711 was placed". A broker is a middle service that holds records between producers and consumers (Kafka, RabbitMQ and Amazon SQS are brokers).
Protobuf / Thrift: Binary formats where fields are identified by number.
Pushdown: Sending a filter or calculation to where the data lives.Sending the filter or the calculation to where the data lives, instead of bringing all the data to where the calculation runs. "Ask the librarian to find the three books" instead of "carry the whole library to your desk".
Rate limit: A cap on how many requests a caller may make per unit of time.
Registry (schema registry): An external service holding schemas, referenced by a small ID in each message.
REST: An API style based on stateless requests and fixed response shapes.
Retry: If an ack does not arrive within some time, the sender gives up waiting (a timeout) and often sends again (a retry). Retrying is useful, but it can create duplicates, as you will see.
Retry / retry storm: Sending a request again; many retries at once can overload a slow service.
Reverse ETL: Pushing data from the warehouse into operational systems.
Row-oriented: Storing each record's fields together, record after record. Suits transactional workloads that touch whole rows; analytical scans of a few columns prefer column layout.Row layout stores each record together: [id, name, price], then the next record. Column layout stores all id values together, then all name values, then all price values. Column layout is good when a query needs only a few columns. It is also good for compression, because one column holds similar values.
S3-compatible API: The object-storage API that became a de facto standard.
Schema: The description of the shape of a record: fields, types, optionality.A description of the shape of a record: which fields (also called columns) exist, what type each has (integer, text, timestamp, ...), and which ones may be missing. Example: a shipment event has shipment_id (text), status (text), weight (number), occurred_at (timestamp).
Self-describing: Carrying its own schema in the file or message.A file or message that carries its own schema, so the reader needs nothing else.
Sentinel file: A marker file written after the data to signal completeness.
Serialization: Turning in-memory values into a flat sequence of bytes.Programs hold values in memory (objects, structs, lists). To send them over a network or store them in a file, you must turn them into a flat sequence of bytes. This is serialization; the reverse is deserialization (or decoding). JSON is a serialization format. So are Protocol Buffers (Protobuf), Avro and Thrift.
Splittability: Whether a reader can start at a random offset and find the next record boundary.
Statelessness: No request depends on an earlier one, so any server can answer any request.
Table format: A specified metadata protocol that defines a table as a set of files plus commits.Object storage (like Amazon S3) stores whole files called objects behind HTTP requests. Parquet is a file format that stores data column by column. A table format (such as Iceberg or Delta) records, in small metadata files called manifests, which data files make up a table. A snapshot is one exact version of the table, identified by a snapshot id. A catalog is the service that tells you the current snapshot of a table.
Throughput: Work completed per unit of time.Throughput is work finished per second. Latency is how long one operation takes. Little's Law links them to the number of operations in progress: L = λ × W, where λ is the arrival rate (per second), W is the average time one operation spends in the system, and L is how many are inside the system at the same time. Example: 1,000 requests per second, each taking 20 ms (0.02 s), means 1,000 × 0.02 = 20 requests in flight at any moment. This is an identity, not a design choice.
Total order: An ordering in which any two values can be compared.A total order means any two values can be compared and one comes first (numbers, dates, strings in alphabetical order). A range is "everything between A and B" in that order. An ordinary database index, the B-tree, keeps keys sorted so that "find everything between A and B" reads one contiguous stretch. A hash index answers only "is this exact value present?" in constant time.
Unload: The engine writes results as files to shared storage and gives the client a pointer.
Webhook: A push in which the producer calls a URL supplied by the consumer.
Zero-copy: Using bytes directly as in-memory values, with no decode step.1 other meaning →
Zero-copy data sharing: Sharing a catalog-mediated reference to data instead of a copy; tighter coupling than copying.