Glossary

88 terms, in the sense this mechanism uses them. The platform glossary has every term across all mechanisms.

Alert fatigue: People ignoring alerts because too many are false. It is a designed-in result of large check suites.
Allowed lateness: How long after a window ends late records are still accepted and the window can fire again.
Andon cord: A factory device that lets any worker stop the line; an analogy that does not transfer to data pipelines.
Anti-entropy: A background process that compares replicas and repairs differences; the only repair method that guarantees convergence.
Batch: Processing a whole window of data at once.A batch job reads a fixed, finished set of data (yesterday's files). A stream job reads data that keeps arriving and never ends. Both can be asked the same question ("how many deliveries per hour?"), but the stream has to answer before it has seen all the data.
Big-O notation: O(files) means "the work grows in proportion to the number of files." O(1) means "the work is the same however many files exist."
Block: Publish nothing when a check fails.1 other meaning →
Blocking: The cheap step in fuzzy matching that decides which pairs will be compared.1 other meaning →
Bloom filter: A bit array and k hash functions answering "definitely not present" or "probably present"A tiny structure that answers "is this key in the set?" with either "definitely not" or "probably yes". It never misses a real member, but sometimes says "probably yes" for a key that is not there (a false positive). It needs about 10 bits per key.
Bulkhead: A partition of resources so a failure in one part cannot consume the resources of another.
Canonicalisation: Converting values to one standard form before comparing; every rule hides a class of real differences.
Check constraint: A database rule that every row must satisfy a predicate.Rules a database can enforce. A primary key says each row has a unique, non-null identifier. A unique constraint says no two rows share a value. A foreign key says a value must exist in another table (for example every orders.customer_id must exist in customers). A check constraint says a row must satisfy a predicate.
Check digit: A checksum inside an identifier; confirms form, not existence.
Circuit breaker: A mechanism that stops calls to a failing dependency (closed, open, half-open).
Comparator: The expensive step in fuzzy matching that scores a candidate pair.
Compare-and-swap (CAS): An atomic instruction that updates a variable only if it still has the expected old value.A lock or latch lets only one thread enter a piece of code or data at a time (a latch is a short lock inside a database). CAS (compare-and-swap) is one CPU instruction: "set this variable to new only if it still equals old". It either succeeds or fails and never blocks. Lock-free algorithms are built from it.
Connected component: A maximal set of vertices that can reach each other.
Consumer: A program that receives and processes records.A producer is a program that creates data and sends it somewhere. A consumer is a program that receives it and does something with it. A record (or message, or event) is one unit of data: for example, "order 4711 was placed". A broker is a middle service that holds records between producers and consumers (Kafka, RabbitMQ and Amazon SQS are brokers).
Contract (data): An agreement about meaning, owner, consumers, freshness and change process for a data product.
DAG: Directed acyclic graph: tasks joined by arrows with no loops.Directed acyclic graph: boxes (tasks) joined by arrows (dependencies) with no loops. "Directed" means each arrow has a direction. "Acyclic" means you cannot follow the arrows and come back to where you started. An arrow A → B means "B may start after A finishes."
Data diff: Running old and new code on the same input and comparing the two outputs.
Data pipeline: A chain of jobs that moves data from where it is created to where it is read, each job reading inputs and writing outputs; usually drawn as a DAG of steps. It runs in batches or record by record (streaming).A chain of jobs that moves data from where it is created (an app, a partner's file, a payment provider) to where it is read (a report, a dashboard). Each job reads input and writes output. The chain is often drawn as a DAG (directed acyclic graph: steps with arrows, and no loops). Freshlane's pipeline is a DAG of 14 steps. A batch pipeline processes a whole day (or hour) of data at once. A streaming pipeline processes each record as it arrives.
Dead-letter queue (DLQ): A holding area for records rejected by validation.A place where a pipeline puts records it could not process or did not accept, so they are kept instead of lost.
Defect cost: Roughly detection latency × downstream fan-out × consequence per consumer.
Defect rate: The fraction of records that are wrong. "10⁻⁴" means 1 in 10,000. "10⁻⁶" means 1 in a million.
Differential: A comparison of two answers; detects difference, not blame.
Drift: Unplanned change in the shape or types of incoming data.
Dual-write: Writing both the old and the new field during expand/contract.
Effect size: How large the difference is, as opposed to whether it is statistically detectable.
Entity resolution: Deciding that several records describe the same real-world entity when no shared key exists.
Exchange (shuffle): Sending rows over the network so equal keys meet on one machine.
False negative: A check that stays silent although the data is bad.A check can be wrong in two ways. A false positive (a false alarm) fires when the data is fine. A false negative (a miss) stays silent when the data is bad. Precision is: of the alarms that fired, what fraction were real defects. Recall is: of the real defects, what fraction did an alarm catch. You cannot push both false positives and false negatives to zero at the same time. You move a threshold and buy one by paying with the other.
False positive: A check that fires although the data is fine.A check can be wrong in two ways. A false positive (a false alarm) fires when the data is fine. A false negative (a miss) stays silent when the data is bad. Precision is: of the alarms that fired, what fraction were real defects. Recall is: of the real defects, what fraction did an alarm catch. You cannot push both false positives and false negatives to zero at the same time. You move a threshold and buy one by paying with the other.
Fan-out: The number of consumers (or nodes) a piece of data reaches.1 other meaning →
Foreign key: Constraint that every value in a column exists in a column of another table.Rules a database can enforce. A primary key says each row has a unique, non-null identifier. A unique constraint says no two rows share a value. A foreign key says a value must exist in another table (for example every orders.customer_id must exist in customers). A check constraint says a row must satisfy a predicate.
Freshness: now − max(event_time) in a dataset.
Gate: A check with authority to stop publication.
Golden fixture: A frozen input and expected output used in tests.
Ground truth: The referent of a statement is the thing in the world the statement is about. "This customer ID is real" is about customers, not about bytes. A system has ground truth for a statement if it can settle the statement from information it already holds.
Hash: A hash function turns any input into a fixed-size fingerprint. The same input always gives the same fingerprint. It is meant to be one-way: from the fingerprint you should not be able to recover the input.
HyperLogLog: A distinct-count sketch that keeps the longest leading-zero run per bucket; error 1.04/√m.
Idempotent: Doing it twice has the same effect as doing it once.An operation is idempotent if doing it twice has the same effect as doing it once. "Set the order status to SHIPPED" is idempotent. "Add 1 to the stock counter" is not. Idempotence is the main tool for living with duplicates.
Inferred member: A synthetic dimension value such as "Unknown" given to a fact row with no match, so the defect is counted.
Informational constraint: A constraint declared but not enforced; the optimizer may still trust it.
Join: Combining rows of two tables where a condition holds, such as attaching each order to its customer. An equijoin matches on equality; a range join uses <, > or BETWEEN.Combining rows of two tables where a condition holds. orders JOIN customers ON orders.customer_id = customers.customer_id attaches each order to its customer. In this mechanism we call the two inputs R and S, with sizes |R| and |S|. An equijoin uses =. A join that uses <, > or BETWEEN is a range or inequality join.
KLL / t-digest: Sketches that estimate quantiles in bounded memory.
Kolmogorov–Smirnov (KS) test: A significance test comparing two distributions.
Lateness: In pipelines, a watermark is a statement "all data older than time T has now arrived". Allowed lateness is how long you wait for stragglers before you treat a time window as complete.
Lineage: The map of which assets feed which.A map of what feeds what: table A is read by job J, which writes table B. Following the map shows what would be affected if A changes.
LSH (locality-sensitive hashing): Hashing designed so that similar items collide on purpose.
Merkle tree: A tree of hashes used to find which ranges of two datasets differ.
Metastable failure: A failure that persists after its trigger is gone because retries keep the system overloaded.
MinHash: A signature whose match fraction estimates Jaccard similarity.
Monetary-unit sampling: Sampling in proportion to value, not row count.
Multiple comparisons: The effect that many tests produce chance "failures"; the root of alert fatigue.
Null: A present field that has no value.In SQL, NULL means "unknown". A condition can be TRUE, FALSE or UNKNOWN. NULL = NULL is UNKNOWN, not TRUE. A WHERE clause keeps only rows where the condition is TRUE. This small fact causes many of the corner cases here.
Optimizer: The database part that chooses a query plan using statistics and declared facts.The part of a database that chooses how to run a query. It uses statistics (row counts, distinct counts, min and max) and declared facts (such as "this column is unique") to choose a fast plan.
Overlay (entity map): A mapping from source records to entity IDs that leaves source data untouched.
Percolation point: The link density at which a giant connected group suddenly appears.
Phonetic code: A code giving equal values to words that sound alike.
Poison pill: A record that fails deterministically on every retry and can block an ordered partition.
Precision: Of the alarms that fired, the fraction that were real defects.Of the results returned, precision is the fraction that is correct. Of all the correct answers that exist, recall is the fraction you returned. If 10 items are truly relevant and you return 8 of them among 20 results, recall = 8/10 = 80% and precision = 8/20 = 40%.
Predicate: The condition in a query that decides which records qualify.The condition in a query that decides which records qualify. In WHERE price < 500 AND city = 'Pune', the predicate is price < 500 AND city = 'Pune'.
Primary key: A unique, non-null identifier for each row.Rules a database can enforce. A primary key says each row has a unique, non-null identifier. A unique constraint says no two rows share a value. A foreign key says a value must exist in another table (for example every orders.customer_id must exist in customers). A check constraint says a row must satisfy a predicate.
Producer: A program that creates and sends records.A producer is a program that creates data and sends it somewhere. A consumer is a program that receives it and does something with it. A record (or message, or event) is one unit of data: for example, "order 4711 was placed". A broker is a middle service that holds records between producers and consumers (Kafka, RabbitMQ and Amazon SQS are brokers).
PSI (population stability index): A measure of change between two distributions, independent of row count.
Quantile: The p99 of a set of numbers is the value that 99% of them are below. p99.99 means 99.99% are below. These matter a lot here, because waiting decisions depend on the slowest records, not the average ones.
Quarantine: Setting offending records aside and publishing the rest.
Recall: Of the real defects, the fraction that were caught.Of the results returned, precision is the fraction that is correct. Of all the correct answers that exist, recall is the fraction you returned. If 10 items are truly relevant and you return 8 of them among 20 results, recall = 8/10 = 80% and precision = 8/20 = 40%.
Reconciliation: Comparing data to a second independent source.Checking data against a second, independent source ("our revenue total equals the payment provider's total").
Referent: The thing in the world that a statement is about.The referent of a statement is the thing in the world the statement is about. "This customer ID is real" is about customers, not about bytes. A system has ground truth for a statement if it can settle the statement from information it already holds.
Restate: Publishing a correction over data already released.
Schema: The description of the shape of a record: fields, types, optionality.A description of the shape of a record: which fields (also called columns) exist, what type each has (integer, text, timestamp, ...), and which ones may be missing. Example: a shipment event has shipment_id (text), status (text), weight (number), occurred_at (timestamp).
Schema validation: Runtime checking that a record has the agreed shape.
Set membership: A check that a value is in an allowed list; secretly depends on data.
Sketch (HyperLogLog): A small summary of a data set that answers one kind of question approximately.A quantile is a percentile: the value below which a given fraction of data falls (the median is the 50% quantile). A sketch is a small, fixed-size summary of a huge dataset that answers one kind of question approximately: for example a t-digest or KLL sketch answers quantile questions, HyperLogLog counts distinct values, and Space-Saving finds the most frequent values. A sketch is built in a single pass and uses bounded memory.
Snapshot: A frozen consistent view of the database at one moment.1 other meaning →A lock makes other transactions wait while you use a row. A snapshot is a frozen, consistent view of the database as it was at one moment; with it, readers do not need locks, because they read old versions while writers create new ones. This second idea is called MVCC (multi-version concurrency control): the database keeps several versions of each row. Both are mechanisms. This mechanism is about the promises they are used to keep.
Space-Saving: A heavy-hitter algorithm that evicts the minimum counter and adopts its count plus one.
Stratify: Split data into groups along a dimension and sample each group.
Streaming: Processing each record as it arrives.1 other meaning →
System of record: The system whose answer wins when systems disagree.
Transitivity: If A matches B and B matches C then A matches C; pairwise matching does not guarantee it.
Trust boundary: The edge of the network region you trust without further encryption or authentication.
Unique: A constraint that no two rows share a value in a column or set of columns. A primary key is unique and also non-null.Rules a database can enforce. A primary key says each row has a unique, non-null identifier. A unique constraint says no two rows share a value. A foreign key says a value must exist in another table (for example every orders.customer_id must exist in customers). A check constraint says a row must satisfy a predicate.
Validation: Checking data against a stated expectation.Checking data against a stated expectation ("price is between 1 and 100,000", "end date is not before start date"). A person wrote the expectation down.
Watermark: Declared non-decreasing lower bound on the event times of future arrivals.1 other meaning →Every event carries a timestamp of when it happened (event time), which differs from when the job sees it (processing time). Events arrive late and out of order: a phone with no signal sends its events 5 minutes after they happened. A watermark is a marker flowing through the stream that says "no event with a timestamp earlier than T should arrive any more." It is the job's best guess of how complete the input is. When the watermark passes the end of a time period, the job may treat that period as complete. Watermarks are a guess, and late events can still violate them. The rules of when it is correct to emit a result (event time versus processing time, how late is too late) are a topic of their own. In this mechanism we need only the idea above. We study the machinery that holds data until the watermark allows an answer.
Windowed dedup: De-duplication that only compares records within a time window.
Write-Audit-Publish: Write to staging, check with assertions, then atomically publish.