Glossary

121 terms, in the sense this mechanism uses them. The platform glossary has every term across all mechanisms.

ALP: Float codec that finds decimal exponents so that doubles become exact integers, then uses FOR and bit-packing.
Amplification: Extra bytes read, written or stored compared with what was logically needed.If you ask for 200 bytes and the system must move 4,096 bytes, the amplification is about 20. Read amplification means extra bytes read. Write amplification means extra bytes written. Space amplification means extra bytes stored. It is the hidden cost behind most slow systems.
ANS / FSE: Asymmetric numeral systems and its table-driven form; arithmetic-coding ratio at 1–2 GB/s. Decodes in the opposite order to encoding.
Append-only: A file or log where data is only added at the end.A store where you only add new rows and never change old ones in place. Most analytical stores work this way, because rewriting data in place is expensive on large files.
Arithmetic coding: Entropy coding that writes the whole message as one number in [0,1); near-optimal but slow (about 100–200 MB/s).
Arrow: Standard in-memory column format designed so the CPU can use values with no decode step.
Avro: Row serialization format with zero per-field framing; the reader needs the writer's schema.
Backward compatible: A new reader can read data written by an older writer.
Bandwidth: Bytes (or bits) moved per second.Latency is how long one operation takes before the first byte arrives. Bandwidth (also throughput) is how many bytes arrive per second after that. A truck driving 500 km has a high latency (many hours) and may still carry a huge amount: high bandwidth.
Big-endian: Storing the most significant byte of a number first (network byte order).
Binding (early/late): Fixing a decision; early binding is before seeing the data, late binding is after.
Bit-packing: Storing every value of a block with the same fixed number of bits.
Bits: A bit is a 0 or a 1. A byte is 8 bits and can hold a number from 0 to 255. We write bytes in hexadecimal (base 16): two characters per byte, such as 0x7F (127) or 0xFF (255). The prefix 0x means "hexadecimal".
Blast radius: The part of the system or the number of users affected when one failure happens.How much damage one mistake or one stolen secret can cause.
Block: The unit compressed independently; sets ratio, read amplification, blast radius and parallelism.1 other meaning →Storage hardware and databases do not read single bytes. They read fixed-size chunks called pages (or blocks), typically 4 KiB to 16 KiB on a local disk. To read one 200-byte row you must read its whole page.
Brotli: LZ + entropy codec with a built-in static dictionary of about 13,000 web strings; strong on small web text, slow to decode.
Canonical form: A standard text form of a schema (no whitespace, comments or aliases) used before hashing.
Cascade (of encodings): Applying encodings one after another, for example dictionary then FOR then bit-packing.
Checksum / CRC32C: A short value computed from bytes that changes if any byte changes; CRC32C has a CPU instruction and runs at 10–20 GB/s.
Codepoint: The number Unicode assigns to a character.
Collation key (sort key): A byte string made by a locale library so that byte order equals human sort order; larger and not reversible.
Column layout: Storing all values of one field together rather than one whole record after another. A query that needs a few columns reads only those, which is why analytics formats such as Parquet use it.A row layout stores one whole record after another: [parcel1, status1, weight1], [parcel2, status2, weight2], .... A column layout stores all values of one field together: [parcel1, parcel2, ...], [status1, status2, ...], .... Analytics systems prefer columns because a query usually needs only a few fields. Parquet is a popular column file format. Kafka is a log-style message queue; consumers read messages in order and can replay from the beginning ("offset zero").
Compaction: Rewriting many small files into fewer larger ones.An LSM-tree (Log-Structured Merge tree) is a storage structure that never changes files in place. It collects writes in memory, writes them as sorted, immutable files, and later compacts (merges and rewrites) the files in the background to remove old versions and deleted entries. A tombstone is a marker that says "this key was deleted"; compaction removes the key and the marker. Compaction costs disk and CPU. Write amplification is the ratio of bytes the storage actually writes to bytes the application asked to write. A Bloom filter is a small structure that answers "definitely not in this file" or "maybe in this file", so reads can skip most files.
Compile-time binding: The reader's own code already knows the types, so nothing is sent with the data.
Compression: A program that turns data into fewer bytes. The compression ratio r = original size ÷ compressed size. If 100 MB becomes 25 MB, r = 4. Lossless compression gives back exactly the same bytes; everything here is lossless.
Compression ratio (r): Original size divided by compressed size.
Content-addressing: Identifying something by a hash of its content.
Content-defined chunking: Cutting data at boundaries decided by a rolling hash of the content, so edits move only nearby boundaries.
Context mixing: Very strong but slow compression models (100 KB/s–1 MB/s) that combine many predictions; about 1.1 bits per English character.
Correlated-column detection: Research idea: encoding columns together because they depend on each other (city and postcode); rare in production.
Decode locality: How much of a stream must be touched to recover one value.
Decode throughput (D): Output bytes per second a decoder produces.
Deduplication: Storing each distinct chunk of data once, across files and snapshots.
Definition level: Dremel integer saying how many optional/repeated ancestors of a value are present.
Deflate: The algorithm in gzip and zlib: LZ77 with Huffman coding and a 32 KB window.
Delta / delta-of-delta: Storing the difference to the previous value (or the difference of differences); small numbers, but no random access.
Dictionary encoding: Replacing each value by a small integer index into a table of distinct values.
Dremel: Google system and method for storing nested records in columns using definition and repetition levels.
Endianness: The byte order used to store multi-byte numbers.A multi-byte number can be stored with its most significant byte first (big-endian, also called network byte order) or its least significant byte first (little-endian, what most CPUs use). The number 258 = 0x0102 is stored as 01 02 in big-endian and 02 01 in little-endian.
Entropy: The minimum average bits per symbol needed given only symbol frequencies.1 other meaning →In information theory, the entropy of a source is the minimum average number of bits per symbol needed to describe it, if you only know how often each symbol occurs. We will meet the formula in Part 3.
Entropy coding: Giving frequent symbols fewer bits (Huffman, arithmetic coding, ANS).
Exception path / patch list: A second storage place for values that do not fit the common case (PFOR exceptions, dictionary fallback).
Extent: Where a value starts and stops in a byte stream.
Fail-fast / fail-degraded: Stop on bad input, or continue with reduced correctness.
Fingerprint: A short hash of a key stored instead of the key.
Floating-point numbers: Decimal fractions like 12.37 are stored in the IEEE-754 format: one sign bit, a group of bits for an exponent (the size of the number) and a group for the mantissa (the digits). A 64-bit float ("double") has 1 + 11 + 52 bits.
Footer: Metadata at the end of a file, such as a directory of offsets and statistics.
FOR (frame of reference): Storing a block minimum once and each value as an offset from it; keeps random access.
Forward compatible: An old reader can read new data. Allows writers to be deployed first.
Framing cost: Bytes spent per field or record to mark extent, type or identity.
Front coding (prefix compression): For sorted strings, storing the length of the shared prefix with the previous string plus the different suffix.
Gorilla XOR: Float codec that XORs consecutive values and stores only the non-zero middle bits; works when values change slowly.
Granularity: The smallest increment a mechanism handles or bills.The smallest piece a mechanism handles. If a courier charges per whole kilogram, one kilogram is the granularity, even if you ship 100 grams. When what you need is smaller than the granularity, you pay for the difference. This mechanism calls that difference granularity waste.
Hexadecimal: Base-16 notation; two characters per byte.A bit is a 0 or a 1. A byte is 8 bits and can hold a number from 0 to 255. We write bytes in hexadecimal (base 16): two characters per byte, such as 0x7F (127) or 0xFF (255). The prefix 0x means "hexadecimal".
Huffman coding: Entropy coding with whole-bit codes; wastes up to about one bit on very skewed symbols.
ICU: A widely used library for Unicode and locale-aware text handling, including sort keys.
IEEE-754: Standard format for floating-point numbers (sign, exponent, mantissa).
Indirection: An extra lookup layer between a name and a physical location.A lookup layer between "the name you use" and "where the thing really is", like a phone book or a URL redirect. It lets you move things without telling everyone.
Interleaved streams: Several independent entropy-coded streams decoded together to keep the CPU busy.
Kafka: A log-style message queue where consumers read in order and can replay.
Latency: Time one operation takes from request to response.Throughput is work finished per second. Latency is how long one operation takes. Little's Law links them to the number of operations in progress: L = λ × W, where λ is the arrival rate (per second), W is the average time one operation spends in the system, and L is how many are inside the system at the same time. Example: 1,000 requests per second, each taking 20 ms (0.02 s), means 1,000 × 0.02 = 20 requests in flight at any moment. This is an identity, not a design choice.
LIFO (in ANS): Last-in-first-out: ANS decodes in the reverse direction of encoding, so blocks must be buffered.
Lossless: Compression that restores exactly the original bytes.
LZ4 / Snappy: Very fast LZ codecs without an entropy coder; ratio about 2, decode 1.5–4 GB/s.
LZ77 (LZ family): Compression by replacing repeats with (offset, length) references to earlier data in a window.
Mantissa: The digit bits of a floating-point number.
Match finding: The search for earlier copies of the current text; it decides the compression level.
memcmp: Byte-by-byte, unsigned, lexicographic comparison of two memory regions.
MessagePack: A compact binary encoding that resembles JSON.
Minimum match length: The shortest repeat worth replacing by a reference.
Model (in compression): The probability assumptions a coder uses; for strong compression, the model matters more than the coder.
NVMe: Protocol for fast, highly parallel SSD access over PCIe.
Object storage: A cloud service holding named objects fetched by HTTP, replaced whole, never edited.A cloud service where each file ("object") has a name and is fetched with an HTTP request. A request costs 20-100 ms before the first byte arrives, and the provider charges a small fee per request. You cannot change a few bytes inside an object. You replace the whole object.
Offsets buffer: Arrow buffer where element i occupies bytes offsets[i] to offsets[i+1].
Opaque (data): Data that cannot be indexed into after compression; you must decode a whole block to read one value.
Operability: The ability to compare, filter or aggregate encoded data without fully decoding it.
Order-0 / conditional entropy: Order-0 assumes symbols are independent; conditional entropy uses the previous symbols as context and is lower.
Order-preserving dictionary: A dictionary whose codes sort like the values, so range predicates work on codes; needs two passes.
Order-preserving encoding: Turning typed values into bytes so that byte order equals logical order.
p99: The latency that 99% of requests beat; shows the slow tail.Time 1,000 requests and sort them from fastest to slowest. The p50 (median) is the 500th: half were faster. The p99 is the 990th: only 1% were slower. The p99.9 is the 999th. The slowest requests are called the tail. The mean (average) adds all the times and divides by the count.
Page: Unit of compression in a Parquet file, about 1 MB.3 other meanings →
Parquet: Column-oriented file format with row groups, pages and a footer.Object storage (like Amazon S3) stores whole files called objects behind HTTP requests. Parquet is a file format that stores data column by column. A table format (such as Iceberg or Delta) records, in small metadata files called manifests, which data files make up a table. A snapshot is one exact version of the table, identified by a snapshot id. A catalog is the service that tells you the current snapshot of a table.
PFOR: Patched frame of reference: choose a width for most values and store the rest as exceptions.
Point lookup: Reading one small record by key.
popcount: CPU instruction that counts the 1-bits in a word.
Predicate pushdown: Moving a filter earlier in the plan, closer to the data.
Projection: Reading only the columns a query needs.2 other meanings →
Protobuf: Serialization format with a numbered tag per field; skippable, not self-sufficient.
Repetition level: Dremel integer saying at which repeated ancestor a new list starts.
Restart point: A full (non-delta) value stored every few entries to bound the dependency chain and allow search.
Rolling hash: A hash computed over a sliding window of bytes, cheap to update by one byte.
Row group: A block of rows (for example 128 MB) that is the unit of independent metadata and rewrite.
Row layout: Storing one whole record after another. Good for reading or writing whole records; a query that needs a few columns still reads every field of every row.A row layout stores one whole record after another: [parcel1, status1, weight1], [parcel2, status2, weight2], .... A column layout stores all values of one field together: [parcel1, parcel2, ...], [status1, status2, ...], .... Analytics systems prefer columns because a query usually needs only a few fields. Parquet is a popular column file format. Kafka is a log-style message queue; consumers read messages in order and can replay from the beginning ("offset zero").
Run-length encoding (RLE): Storing (value, count) pairs for runs of equal values.
Sampled cascade search: Choosing encodings by trying candidates on about 1% of the data.
Schema: The description of the shape of a record: fields, types, optionality.A description of the shape of a record: which fields (also called columns) exist, what type each has (integer, text, timestamp, ...), and which ones may be missing. Example: a shipment event has shipment_id (text), status (text), weight (number), occurred_at (timestamp).
Schema registry: A service that stores schemas and returns an ID for each.A small central service that stores every version of every schema and checks new versions against rules. Producers and consumers ask it which schema applies. We study it in Part 5.
Schema-on-read / schema-on-write: Structure is checked when reading, or when writing.
Self-describing format: A format whose bytes carry field names and types (for example JSON).
Self-synchronizing: From any byte one can find where the next character starts (UTF-8).
Sentinel value: A special value (such as −1) used to mean "no value".
Serialization: Turning in-memory values into a flat sequence of bytes.Programs hold values in memory (objects, structs, lists). To send them over a network or store them in a file, you must turn them into a flat sequence of bytes. This is serialization; the reverse is deserialization (or decoding). JSON is a serialization format. So are Protocol Buffers (Protobuf), Avro and Thrift.
Skew: Uneven symbol frequencies, the redundancy that entropy coding removes.2 other meanings →
Skippable: An unknown field can be walked over using its framing.
Static dictionary: A dictionary built into the codec itself (as in Brotli).
Storing a number: A 32-bit integer always uses 4 bytes, a 64-bit integer 8 bytes, whatever the value. Negative numbers use two's complement: to write −1 in 64 bits, all 64 bits are set to 1. Because the top bit is 1 for every negative number, a negative number looks like a huge positive number if you ignore the sign. We will use this fact several times.
Strict (Pareto) domination: Being better on every measure; ZSTD against gzip.
Tail statistic: A measure such as the maximum or the 99th percentile that depends on rare values.
Text encoding: Computers store text as numbers called codepoints (for example "A" is 65). UTF-8 is the common way to write codepoints as bytes: ASCII characters use 1 byte, other characters use 2 to 4 bytes.
Throughput units: MB/s and GB/s mean megabytes and gigabytes per second. Decoding "at 1 GB/s" means the decoder produces 1 GB of output per second on one CPU core.
Trained dictionary: A shared artifact (about 110 KB for ZSTD) built from sample records that pre-fills the window and entropy tables for small inputs.
Transitive compatibility: Checking a new schema against every earlier version, not only the last.
Two's complement: Standard way to store negative integers; −1 is all 1-bits.
UTF-8: Variable-length (1–4 bytes) encoding of Unicode codepoints.
Validity bitmap: One bit per value saying whether the value is present (not null). Compared with a position list, it costs N bits against m·log₂N; crossover at density 1/log₂N (about 5%).
Varint: Variable-length integer: 7 payload bits per byte plus a continuation bit.
Window: How far back a match may reach: 32 KB (deflate), 64 KB (LZ4), 8 MB (ZSTD default at high levels), up to 16 MB (Brotli).2 other meanings →
Wire type: The Protobuf tag bits that say how to read the next value.
Zigzag: Mapping of signed integers to unsigned so small magnitudes stay small.
Zone map: Per-file min and max values of a column, used to skip files.A zone map stores the minimum and maximum of a column for each block of rows. If your range does not overlap a block's min/max, you skip the block without reading it. Partition pruning is the same idea at table level: a table is split into partitions by a key (for example by date), and the engine skips partitions that cannot match. Both rely on values being comparable, so that "min" and "max" mean something.
Zstd: Modern general-purpose compressor with a fast decoder at every level.