Glossary

79 terms, in the sense this mechanism uses them. The platform glossary has every term across all mechanisms.

Accumulation mode: Whether a re-fired window emits the new total (accumulating) or only the change (discarding).
Allowed lateness: How long after a window ends late records are still accepted and the window can fire again.
As-of join: A join that, for each left row at time t, picks the right row with the greatest time at or before t.A join that, for each event at time t, picks the most recent value of another table that was valid at or before t.
Associative: An operation where grouping does not matter, such as sum or max; it allows partial results to be combined.An operation is associative if grouping does not matter: (a + b) + c = a + (b + c). Sum, count, min and max are associative. This means you can add up pieces separately and then add the partial sums. Average and median are not associative in this plain form.
Backfill: Running a workflow for past intervals by iterating over the interval key space.1 other meaning →Going back over existing data to fill in or fix a value. Example: a new column exists, and a backfill job computes it for the last three years of rows.
Batch: Processing a whole window of data at once.A batch job reads a fixed, finished set of data (yesterday's files). A stream job reads data that keeps arriving and never ends. Both can be asked the same question ("how many deliveries per hour?"), but the stream has to answer before it has seen all the data.
Bitemporal: Storing both valid time and transaction time for each fact.
Calendar unit: Month, day or year; not a fixed number of seconds.
Commutative: Two operations give the same result in either order (for example two decrements).A combining operation is associative if grouping does not matter: (a + b) + c = a + (b + c). It is commutative if order does not matter: a + b = b + a. It is invertible if you can undo it: subtract to undo an addition. Sum and count have all three properties. Maximum is associative and commutative but not invertible. Median has none of the three in a useful form. These properties decide how cheaply an aggregate can be computed piece by piece.
Compaction: Rewriting many small files into fewer larger ones.An LSM-tree (Log-Structured Merge tree) is a storage structure that never changes files in place. It collects writes in memory, writes them as sorted, immutable files, and later compacts (merges and rewrites) the files in the background to remove old versions and deleted entries. A tombstone is a marker that says "this key was deleted"; compaction removes the key and the marker. Compaction costs disk and CPU. Write amplification is the ratio of bytes the storage actually writes to bytes the application asked to write. A Bloom filter is a small structure that answers "definitely not in this file" or "maybe in this file", so reads can skip most files.
Consumer: A program that receives and processes records.A producer is a program that creates data and sends it somewhere. A consumer is a program that receives it and does something with it. A record (or message, or event) is one unit of data: for example, "order 4711 was placed". A broker is a middle service that holds records between producers and consumers (Kafka, RabbitMQ and Amazon SQS are brokers).
Count-Min Sketch: A d × w table of counters that estimates item frequencies with overestimation proportional to total stream size.
De-duplication (dedup): Removing records that were delivered or processed more than once. It needs memory of which records were already seen.
Dead-letter queue (DLQ): A holding area for records rejected by validation.A place where a pipeline puts records it could not process or did not accept, so they are kept instead of lost.
Dimensional collapse: Merging two independent time coordinates into one, which loses information irreversibly.
Discarding mode: Re-fires emit only the change since the last fire.
Early / on-time / late trigger: Firing before, at, or after the watermark passes the window end.
Epoch: 1 January 1970 00:00 UTC, the zero point of Unix time.2 other meanings →A timestamp is a number or text that names an instant. UTC is the world's standard reference time. Computers often count seconds since the epoch, which is 1 January 1970 00:00 UTC. A phone that lost its clock may report the epoch itself ("1970-01-01"), which is a very common wrong timestamp.
Event: A small message announcing that something happened.An event is a small record: "order 77 placed by user 12 at 12:03:04". A log is an append-only file or service: new events are added at the end, and old events are never changed. A topic is a named log (the word comes from systems like Apache Kafka). Every event has an offset, its position in the log (0, 1, 2, …). A reader remembers "I have read up to offset 5,000", and can go back to offset 3,000 and read again. This ability to replay is the foundation of this whole mechanism. A log keeps events for a configured time (retention), for example 7 days, then deletes the oldest ones.
Event time: When something happened in the world, stamped near the source.Every event carries a timestamp of when it happened (event time), which differs from when the job sees it (processing time). Events arrive late and out of order: a phone with no signal sends its events 5 minutes after they happened. A watermark is a marker flowing through the stream that says "no event with a timestamp earlier than T should arrive any more." It is the job's best guess of how complete the input is. When the watermark passes the end of a time period, the job may treat that period as complete. Watermarks are a guess, and late events can still violate them. The rules of when it is correct to emit a result (event time versus processing time, how late is too late) are a topic of their own. In this mechanism we need only the idea above. We study the machinery that holds data until the watermark allows an answer.
Event-time lag: Now − largest observed event time: how stale the input is.
Exactly-once: Each message's effect happens once; real only within one system's transaction boundary.A processing guarantee that each input affects the result exactly one time, even when machines crash and retry. It is hard; we will only need the two basic ideas (idempotence, and committing a position together with the state) when it appears.
Future leakage: Using information in a row that did not exist at the event's time.
Group (algebra): A monoid in which every change can be undone (an inverse exists); needed to support deletions incrementally.
Heavy tail: A distribution in which a small fraction of items carry most of the total size.A distribution with a few extremely large values (a phone offline for a week) far above the typical ones.
Heuristic watermark: Max observed event time − δ.
HyperLogLog: A distinct-count sketch that keeps the longest leading-zero run per bucket; error 1.04/√m.
Idempotent: Doing it twice has the same effect as doing it once.An operation is idempotent if doing it twice has the same effect as doing it once. "Set the order status to SHIPPED" is idempotent. "Add 1 to the stock counter" is not. Idempotence is the main tool for living with duplicates.
Idle source: A partition that sends nothing, which can freeze the global watermark.
Idleness timeout: Rule that ignores a silent partition when computing the watermark.
Incremental view maintenance (IVM): Updating a saved query result from the change to its input, without recomputing it.
Ingestion time: When the record entered your trust boundary (broker or landing zone).
Instant: A precise point in physical time, independent of time zones.
Interval join: A join with a time bound between the two sides; the only kind of stream-stream join that can discard state.
Keyed: A table is keyed if each row has a key (such as (depot, hour)) that identifies it, so a row can be found and overwritten.
Lazy / eager / bulk expiry: Expire on access / with a timer per key / during a scan or compaction pass.
Leap second: An extra second added to UTC; some operators smear it over a window.
Lease: A lock that expires after a time limit unless renewed.
Log position (offset): Gap-free, increasing sequence number assigned inside a partition.
Monoid: An associative operation with an identity element (no inverse required).
Monotonic clock: Clock that never goes backwards; used for durations.A wall clock tells the calendar date and time of day. A monotonic clock only counts elapsed time since some start point and never goes backwards. Part 6 explains why both exist.
NTP: Network protocol that sets computer clocks.
Offset: A consumer's position in a partition (one integer).An event is a small record: "order 77 placed by user 12 at 12:03:04". A log is an append-only file or service: new events are added at the end, and old events are never changed. A topic is a named log (the word comes from systems like Apache Kafka). Every event has an offset, its position in the log (0, 1, 2, …). A reader remembers "I have read up to offset 5,000", and can go back to offset 3,000 and read again. This ability to replay is the foundation of this whole mechanism. A log keeps events for a configured time (retention), for example 7 days, then deletes the oldest ones.
Out-of-order: Records arriving in a different order than their event times.
Partition: One ordered slice of a topic that can be stored and read in parallel.3 other meanings →A topic is a named stream of records ("orders"). Big topics are cut into partitions: separate ordered lists that can be stored and read in parallel. Records in one partition have a fixed order. Across partitions there is no shared order.
Percentile (p50, p99, p99.9): The value below which that fraction of observations fall.p50 is the median latency, p99 is the latency that 99% of requests beat. The slowest requests are the tail. When one request needs answers from many servers (a fan-out) it must wait for the slowest one. With 100 servers that each have a 1% chance of being slow, the chance that at least one is slow is 1 − 0.99¹⁰⁰ = 63%. So the user-facing tail is much worse than any single server's tail.
Perfect watermark: A watermark backed by a proof of the source's ordering.
Processing time: The clock of the machine doing the computation.
Quantile: The p99 of a set of numbers is the value that 99% of them are below. p99.99 means 99.99% are below. These matter a lot here, because waiting decisions depend on the slowest records, not the average ones.
Record: One unit of data sent from a producer to a consumer, such as "order 4711 was placed"; also called a message or an event. A broker holds records between the two.A producer is a program that creates data and sends it somewhere. A consumer is a program that receives it and does something with it. A record (or message, or event) is one unit of data: for example, "order 4711 was placed". A broker is a middle service that holds records between producers and consumers (Kafka, RabbitMQ and Amazon SQS are brokers).
Reorder buffer: A buffer (such as a heap) used to put disordered records into order.
Replay determinism: The property that re-running a computation gives identical results.
Retraction: Removing the effect of an earlier input, expressed as a −1 change.
SCD Type 2: Warehouse pattern that adds a row with start and end dates for each change; valid time only.
Side output: A separate output for records that do not fit the main path, such as late records.
Sink: Where a pipeline writes its results.A sink is where a pipeline writes its output (a table, a dashboard, a message queue). A consumer is whoever reads that output.
Skew: Processing time − event time for a record.2 other meanings →
SLA: Service-level agreement: a promise, here usually about the timing or freshness of data.An SLI (service level indicator) is a measured quantity, such as "fraction of requests that succeed". An SLO (objective) is the target for it, such as "99.9% over 30 days". An SLA (agreement) is an SLO written into a contract, usually with penalties.
Smearing: Spreading a leap second over a long window of time.
Snapshot isolation (SI): Reads from a snapshot at start plus a commit-time check of write sets; permits write skew.
State: Memory a program keeps between records.To compute a running count the program must remember something between records. This memory is called state. State costs memory or disk, and a large part of this mechanism is about how long state must be kept.
Stream: Data that keeps arriving and never ends, processed as it comes. A stream job must answer before it has seen all the data, unlike a batch job over a finished data set.A batch job reads a fixed, finished set of data (yesterday's files). A stream job reads data that keeps arriving and never ends. Both can be asked the same question ("how many deliveries per hour?"), but the stream has to answer before it has seen all the data.
t-digest: A quantile sketch with small clusters near the extremes, giving good tail accuracy without proofs.
Time travel: Reading a table as it was at an earlier moment (transaction time at storage level).
Timestamp: A value that names an instant, usually as seconds since the Unix epoch (1 January 1970, UTC). A device that lost its clock often reports the epoch itself, a common wrong timestamp.A timestamp is a number or text that names an instant. UTC is the world's standard reference time. Computers often count seconds since the epoch, which is 1 January 1970 00:00 UTC. A phone that lost its clock may report the epoch itself ("1970-01-01"), which is a very common wrong timestamp.
Training-serving skew: Any difference between the feature values seen in training and in serving.
Transaction time: When the system recorded or believed a fact.
Trigger: Rule that decides when a window's result is emitted.1 other meaning →
TTL: Time to live: how long an entry may be served before it expires.A lifetime attached to a cache entry. After the TTL expires, the entry is dropped or treated as missing.
Tumbling window: A window that cuts time into disjoint, equal buckets.
Upsert: Insert-or-replace by key.A primary key is a column (or set of columns) whose value is unique per row. An upsert means "insert this row, or update it if the key already exists". In SQL: INSERT ... ON CONFLICT (order_id) DO UPDATE ....
UTC: Coordinated Universal Time, the world's reference time. Timestamps stored in UTC compare correctly across time zones; local time is for display.A timestamp is a number or text that names an instant. UTC is the world's standard reference time. Computers often count seconds since the epoch, which is 1 January 1970 00:00 UTC. A phone that lost its clock may report the epoch itself ("1970-01-01"), which is a very common wrong timestamp.
Valid time: When a fact was true in the world.
Wall clock: A wall clock tells the calendar date and time of day. A monotonic clock only counts elapsed time since some start point and never goes backwards. Part 6 explains why both exist.
Watermark: Declared non-decreasing lower bound on the event times of future arrivals.1 other meaning →Every event carries a timestamp of when it happened (event time), which differs from when the job sees it (processing time). Events arrive late and out of order: a phone with no signal sends its events 5 minutes after they happened. A watermark is a marker flowing through the stream that says "no event with a timestamp earlier than T should arrive any more." It is the job's best guess of how complete the input is. When the watermark passes the end of a time period, the job may treat that period as complete. Watermarks are a guess, and late events can still violate them. The rules of when it is correct to emit a result (event time versus processing time, how late is too late) are a topic of their own. In this mechanism we need only the idea above. We study the machinery that holds data until the watermark allows an answer.
Watermark lag: Now − watermark: how stale the completeness claim is.
Window: A way of cutting an endless stream into finite pieces by time.2 other meanings →A way to cut an endless stream into finite pieces by time. "Orders per restaurant in each 10-minute period" is a window computation. Windows are explained fully in Part 3.
δ (delta): The amount subtracted from the maximum observed event time to form a heuristic watermark.
δ-CRDT: A state-based CRDT that ships only the recent change instead of the whole state.