Glossary

99 terms, in the sense this mechanism uses them. The platform glossary has every term across all mechanisms.

After-match skip strategy: In pattern matching, the rule for where matching continues after a match is found; it decides how many overlapping matches survive.
Aligned barrier: A checkpoint method where an operator waits for the barrier on all inputs before taking its snapshot.
Amplification: Extra bytes read, written or stored compared with what was logically needed.If you ask for 200 bytes and the system must move 4,096 bytes, the amplification is about 20. Read amplification means extra bytes read. Write amplification means extra bytes written. Space amplification means extra bytes stored. It is the hidden cost behind most slow systems.
Associative: An operation where grouping does not matter, such as sum or max; it allows partial results to be combined.An operation is associative if grouping does not matter: (a + b) + c = a + (b + c). Sum, count, min and max are associative. This means you can add up pieces separately and then add the partial sums. Average and median are not associative in this plain form.
Async I/O: Allowing many remote requests in flight at once, so throughput is not limited to 1 ÷ RTT.
Backpressure: A signal from a consumer, based on its actual state, that makes the producer slow down.If a slow step cannot keep up, it tells the step before it to slow down, and that step tells the one before it, until the source stops reading. We study this in Part 8.
Barrier: A marker injected into the stream that tells each operator when to snapshot its state.1 other meaning →
Batch processing: A job that reads a fixed input, produces an output and ends.
Bitemporal: Storing both valid time and transaction time for each fact.
Bloom filter: A bit array and k hash functions answering "definitely not present" or "probably present"A tiny structure that answers "is this key in the set?" with either "definitely not" or "probably yes". It never misses a real member, but sometimes says "probably yes" for a key that is not there (a false positive). It needs about 10 bits per key.
Broadcast state: A small table or rule set copied to every instance.
Buffer timeout: Maximum time a network buffer waits before being sent, even if it is not full.
CEP (complex event processing): Pattern matching over sequences of events.
Change data capture (CDC): Turning a database's changes into a stream of records.A feed of every change (insert, update, delete) made to a database table, delivered as a stream of events.
Changelog: A stream of changes (upserts and deletes) that, when applied in order, produces a table.
Channel: A connection carrying records between two operator instances.
Checkpoint: A saved snapshot of all operator state plus source offsets.1 other meaning →A saved snapshot of a job's state plus its position in the input, written to durable storage (storage that survives a crash). After a crash the job loads the last checkpoint and continues from there.
Commutative: Two operations give the same result in either order (for example two decrements).A combining operation is associative if grouping does not matter: (a + b) + c = a + (b + c). It is commutative if order does not matter: a + b = b + a. It is invertible if you can undo it: subtract to undo an addition. Sum and count have all three properties. Maximum is associative and commutative but not invertible. Median has none of the three in a useful form. These properties decide how cheaply an aggregate can be computed piece by piece.
Compacted log: A log in which only the newest entry per key is kept.
Compaction: Rewriting many small files into fewer larger ones.An LSM-tree (Log-Structured Merge tree) is a storage structure that never changes files in place. It collects writes in memory, writes them as sorted, immutable files, and later compacts (merges and rewrites) the files in the background to remove old versions and deleted entries. A tombstone is a marker that says "this key was deleted"; compaction removes the key and the marker. Compaction costs disk and CPU. Write amplification is the ratio of bytes the storage actually writes to bytes the application asked to write. A Bloom filter is a small structure that answers "definitely not in this file" or "maybe in this file", so reads can skip most files.
Compaction debt: Compaction work that has fallen behind and starts to stall writes.
Consistent cut: A snapshot in which state reflects exactly a prefix of the input, with matching source offsets.
Credit-based flow control: A scheme where the consumer gives the producer credits for the buffers it can accept.
Disaggregated state: State stored in object storage or a shared service, separate from the compute node.
Duality (table-stream): A table is the accumulation of a changelog; a changelog is the sequence of changes of a table.
Enrichment: Adding fields from a reference table to each stream record.
Event: A small message announcing that something happened.An event is a small record: "order 77 placed by user 12 at 12:03:04". A log is an append-only file or service: new events are added at the end, and old events are never changed. A topic is a named log (the word comes from systems like Apache Kafka). Every event has an offset, its position in the log (0, 1, 2, …). A reader remembers "I have read up to offset 5,000", and can go back to offset 3,000 and read again. This ability to replay is the foundation of this whole mechanism. A log keeps events for a configured time (retention), for example 7 days, then deletes the oldest ones.
Event time: When something happened in the world, stamped near the source.Every event carries a timestamp of when it happened (event time), which differs from when the job sees it (processing time). Events arrive late and out of order: a phone with no signal sends its events 5 minutes after they happened. A watermark is a marker flowing through the stream that says "no event with a timestamp earlier than T should arrive any more." It is the job's best guess of how complete the input is. When the watermark passes the end of a time period, the job may treat that period as complete. Watermarks are a guess, and late events can still violate them. The rules of when it is correct to emit a result (event time versus processing time, how late is too late) are a topic of their own. In this mechanism we need only the idea above. We study the machinery that holds data until the watermark allows an answer.
Event time / processing time: When an event happened, versus when the job sees it.
Evictor: A component that removes elements from a window; it forces raw elements to be retained.
Exactly-once effect: Observable state changes once even though the message is delivered several times.
Fusion (operator fusion): Merging consecutive operators into one task to avoid network hops.
Garbage collection (GC): Runtime component that finds unreachable objects and reclaims their memory.In Java, Go, C# and similar languages you never call free. A background component, the garbage collector, finds objects that nothing refers to any more and reclaims their memory. A tracing collector does this by starting from known roots (variables) and following every reference to mark the objects that are still live. The live set is the total size of all live objects. A generational collector splits the heap into a young generation (new objects, collected often and cheaply) and an old generation (objects that survived several collections). Pauses are moments when the program is stopped so the collector can work.
Global window: A window that never ends, usually combined with a custom trigger.
HyperLogLog: A distinct-count sketch that keeps the longest leading-zero run per bucket; error 1.04/√m.
Idempotent: Doing it twice has the same effect as doing it once.An operation is idempotent if doing it twice has the same effect as doing it once. "Set the order status to SHIPPED" is idempotent. "Add 1 to the stock counter" is not. Idempotence is the main tool for living with duplicates.
Idle channel: An input with no data, excluded from the watermark minimum after a timeout.
Incremental checkpoint: Continuous flushing so the checkpoint position moves smoothly.
Indirection: An extra lookup layer between a name and a physical location.A lookup layer between "the name you use" and "where the thing really is", like a phone book or a URL redirect. It lets you move things without telling everyone.
Interval join: A join with a time bound between the two sides; the only kind of stream-stream join that can discard state.
Invertible: An operation that can be undone (subtraction undoes addition).A combining operation is associative if grouping does not matter: (a + b) + c = a + (b + c). It is commutative if order does not matter: a + b = b + a. It is invertible if you can undo it: subtract to undo an addition. Sum and count have all three properties. Maximum is associative and commutative but not invertible. Median has none of the three in a useful form. These properties decide how cheaply an aggregate can be computed piece by piece.
Key: The field that routes a record, such as user_id. The engine hashes it so that all records with the same key go to the same parallel instance; that redistribution is a shuffle.2 other meanings →To handle many events per second, each operator runs as several copies called instances or tasks. The parallelism is the number of copies. Records usually carry a key (such as user_id). The engine sends all records with the same key to the same instance, by computing hash(key) (a function that turns any key into a number) and using it to pick the instance. This redistribution step is called a shuffle or keyBy.
Key group: A fixed bucket of keys; instances own contiguous ranges of key groups.
Keyed state: State partitioned by key, with exactly one writer per key.
KLL: A mergeable quantile sketch with proven rank error.
Lag: How far the job's read position is behind the end of the log.
Late event: An event that arrives after the watermark has passed its timestamp.
Latency: Time one operation takes from request to response.Throughput is work finished per second. Latency is how long one operation takes. Little's Law links them to the number of operations in progress: L = λ × W, where λ is the arrival rate (per second), W is the average time one operation spends in the system, and L is how many are inside the system at the same time. Example: 1,000 requests per second, each taking 20 ms (0.02 s), means 1,000 × 0.02 = 20 requests in flight at any moment. This is an identity, not a design choice.
Little's Law: L = λW: average items in a system = arrival rate × average time in the system.Throughput is work finished per second. Latency is how long one operation takes. Little's Law links them to the number of operations in progress: L = λ × W, where λ is the arrival rate (per second), W is the average time one operation spends in the system, and L is how many are inside the system at the same time. Example: 1,000 requests per second, each taking 20 ms (0.02 s), means 1,000 × 0.02 = 20 requests in flight at any moment. This is an identity, not a design choice.
Log (append-only log): A stored ordered sequence of records; consumers track their own offset.An event is a small record: "order 77 placed by user 12 at 12:03:04". A log is an append-only file or service: new events are added at the end, and old events are never changed. A topic is a named log (the word comes from systems like Apache Kafka). Every event has an offset, its position in the log (0, 1, 2, …). A reader remembers "I have read up to offset 5,000", and can go back to offset 3,000 and read again. This ability to replay is the foundation of this whole mechanism. A log keeps events for a configured time (retention), for example 7 days, then deletes the oldest ones.
LSM-tree: Log-structured merge tree: buffer writes in memory, flush sorted runs, merge in the background.An LSM-tree (Log-Structured Merge tree) is a storage structure that never changes files in place. It collects writes in memory, writes them as sorted, immutable files, and later compacts (merges and rewrites) the files in the background to remove old versions and deleted entries. A tombstone is a marker that says "this key was deleted"; compaction removes the key and the marker. Compaction costs disk and CPU. Write amplification is the ratio of bytes the storage actually writes to bytes the application asked to write. A Bloom filter is a small structure that answers "definitely not in this file" or "maybe in this file", so reads can skip most files.
max_parallelism: The fixed number of key groups; it caps how far a job can ever scale.
Mergeable: Partial results can be combined into one result; needed for panes and sessions.
Micro-batch: Execution model that processes records in small time-based batches.
NFA: Non-deterministic finite automaton: a state machine that can be in several states at once; used for pattern matching.
Offset: A consumer's position in a partition (one integer).An event is a small record: "order 77 placed by user 12 at 12:03:04". A log is an append-only file or service: new events are added at the end, and old events are never changed. A topic is a named log (the word comes from systems like Apache Kafka). Every event has an offset, its position in the log (0, 1, 2, …). A reader remembers "I have read up to offset 5,000", and can go back to offset 3,000 and read again. This ability to replay is the foundation of this whole mechanism. A log keeps events for a configured time (retention), for example 7 days, then deletes the oldest ones.
Operator: One step of a query plan: scan, filter, project, join, aggregate. Rows flow from the scans at the bottom of the plan to the final operator at the top.A SQL query is turned into a plan: a tree or graph of steps. Each step is an operator: scan (read a table), filter (keep rows that match a condition), project (compute columns), join (combine two tables), aggregate (sum, count, group). A tuple is one row as seen by an operator. Data flows from the scanning operators at the bottom to the final operator at the top.
Operator state: State owned by an operator instance, with no key (offsets, sink handles).
Pane: A small time slice (one slide long) holding a partial aggregate for a sliding window.
Parallelism: The number of parallel instances of an operator.To handle many events per second, each operator runs as several copies called instances or tasks. The parallelism is the number of copies. Records usually carry a key (such as user_id). The engine sends all records with the same key to the same instance, by computing hash(key) (a function that turns any key into a number) and using it to pick the instance. This redistribution step is called a shuffle or keyBy.
Priority queue: A structure that always returns the earliest item first; used by the timer service.
Processing-time join: A join against whatever the other side holds right now; not reproducible on replay.
Queue: A line of work waiting for its turn, like people waiting at a bank counter. A scheduler is the part of a system that decides who is served next.1 other meaning →
Replay: Re-reading records that were already processed.
Retention: How long a log keeps records before deleting them.
Retention rule: The rule that says when state may be discarded; unbounded state means a missing rule.
Retraction: Removing the effect of an earlier input, expressed as a −1 change.
Salting: Adding a random suffix to a hot key so it splits into k sub-keys.
Savepoint: A deliberately taken, stable, self-contained checkpoint owned by the operator of the job.1 other meaning →
Schema: The description of the shape of a record: fields, types, optionality.A description of the shape of a record: which fields (also called columns) exist, what type each has (integer, text, timestamp, ...), and which ones may be missing. Example: a shipment event has shipment_id (text), status (text), weight (number), occurred_at (timestamp).
Schema evolution: Changing the structure of records or state over time.
Serialization: Turning in-memory values into a flat sequence of bytes.Programs hold values in memory (objects, structs, lists). To send them over a network or store them in a file, you must turn them into a flat sequence of bytes. This is serialization; the reverse is deserialization (or decoding). JSON is a serialization format. So are Protocol Buffers (Protobuf), Avro and Thrift.
Session window: A window that closes after a gap with no data and merges when a record bridges two sessions.
Shuffle / keyBy: Redistributing records so that all records of one key reach one instance.
Sink: Where a pipeline writes its results.A sink is where a pipeline writes its output (a table, a dashboard, a message queue). A consumer is whoever reads that output.
Sink / source: The operator that writes results out / reads records in.
Sketch (HyperLogLog): A small summary of a data set that answers one kind of question approximately.A quantile is a percentile: the value below which a given fraction of data falls (the median is the 50% quantile). A sketch is a small, fixed-size summary of a huge dataset that answers one kind of question approximately: for example a t-digest or KLL sketch answers quantile questions, HyperLogLog counts distinct values, and Space-Saving finds the most frequent values. A sketch is built in a single pass and uses bounded memory.
Sliding window: A window that moves forward by a slide, so windows overlap.
Source: The step of a streaming job that reads events from outside, such as from a log or topic, and feeds them to the job's operators.A streaming job is a graph of steps. A source reads events from the log. An operator is one processing step (filter, map, count, join). A sink writes results to an outside system (a database, another log). The whole graph is the topology. Records flow along the edges between operators, called channels.
State: Memory a program keeps between records.To compute a running count the program must remember something between records. This memory is called state. State costs memory or disk, and a large part of this mechanism is about how long state must be kept.
State backend: The component that stores operator state (on-heap, off-heap, embedded LSM, remote).
State TTL: Time after which state entries are deleted automatically.
Stateful / stateless: An operator that does / does not remember earlier records.
Temporal join: A join that uses the dimension version valid at the fact's event time.
Throughput: Work completed per unit of time.Throughput is work finished per second. Latency is how long one operation takes. Little's Law links them to the number of operations in progress: L = λ × W, where λ is the arrival rate (per second), W is the average time one operation spends in the system, and L is how many are inside the system at the same time. Example: 1,000 requests per second, each taking 20 ms (0.02 s), means 1,000 × 0.02 = 20 requests in flight at any moment. This is an identity, not a design choice.
Thundering herd: Many clients acting at the same time, such as retrying or refreshing together.
Timer service: The keyed-state priority queue that stores and fires triggers.
Tombstone: A marker record that says a key was deleted; must outlive every older copy.
Topic: A named stream of records.A topic is a named stream of records ("orders"). Big topics are cut into partitions: separate ordered lists that can be stored and read in parallel. Records in one partition have a fixed order. Across partitions there is no shared order.
Topology: The graph of sources, operators and sinks that makes up a job.A streaming job is a graph of steps. A source reads events from the log. An operator is one processing step (filter, map, count, join). A sink writes results to an outside system (a database, another log). The whole graph is the topology. Records flow along the edges between operators, called channels.
Trigger: Rule that decides when a window's result is emitted.1 other meaning →
Tumbling window: A window that cuts time into disjoint, equal buckets.
Two-phase aggregation: Partial aggregation, a shuffle of partials by group key, then a final merge.
Two-phase commit (2PC): Prepare then decide; participants give up the right to abort after voting yes.A way to make a change appear in two places atomically (all or nothing). Phase 1: every participant prepares and promises it can commit. Phase 2: when all have promised, everyone commits. If any participant says no, everyone aborts. Until phase 2, the change stays invisible.
Unaligned barrier: A checkpoint method that snapshots immediately and stores in-flight records.
Upsert: Insert-or-replace by key.A primary key is a column (or set of columns) whose value is unique per row. An upsert means "insert this row, or update it if the key already exists". In SQL: INSERT ... ON CONFLICT (order_id) DO UPDATE ....
Utilization (ρ): The fraction of time a resource is busy.Utilization (ρ, "rho") is the fraction of time a server is busy. For random arrivals, average response time grows like S ÷ (1 − ρ), where S is the service time: 2×S at 50%, 5×S at 80%, 10×S at 90%, 100×S at 99%. Queues are caused by variability meeting load, and the curve has a wall at 100%. The last few percent of utilization cost more than the first eighty.
Watermark: Declared non-decreasing lower bound on the event times of future arrivals.1 other meaning →Every event carries a timestamp of when it happened (event time), which differs from when the job sees it (processing time). Events arrive late and out of order: a phone with no signal sends its events 5 minutes after they happened. A watermark is a marker flowing through the stream that says "no event with a timestamp earlier than T should arrive any more." It is the job's best guess of how complete the input is. When the watermark passes the end of a time period, the job may treat that period as complete. Watermarks are a guess, and late events can still violate them. The rules of when it is correct to emit a result (event time versus processing time, how late is too late) are a topic of their own. In this mechanism we need only the idea above. We study the machinery that holds data until the watermark allows an answer.
Window: A way of cutting an endless stream into finite pieces by time.2 other meanings →A way to cut an endless stream into finite pieces by time. "Orders per restaurant in each 10-minute period" is a window computation. Windows are explained fully in Part 3.