Interview questions
When compression pays
1Compression gave a 4× ratio, but scans on local NVMe got slower. Why?
Compression speeds up a read only if the decoder outruns the storage: decode speed D must exceed B × r / (r − 1), where B is the speed of the source and r the compression ratio. At r = 4 that is 1.33 × B. With NVMe at about 5 GB/s you need about 6.7 GB/s of decoding, and one ZSTD core gives about 1 GB/s.
Read moreWhen does compression pay?
2Is a higher compression ratio always better for query speed?
No. The factor r / (r − 1) is 2 at r = 2, 1.33 at r = 4 and 1.14 at r = 8, so beyond about 4× an extra ratio saves storage cost but almost no read time.
Read moreWhen does compression pay?
3Is ZSTD at level 19 too slow for data that is read often?
Not for reading. Decompression speed is about the same at every level; only compression is 50–100× slower at high levels. Choose the level from your write-side CPU budget and how many times the data will be read.
Read moreZSTD level and decode speed
4Why did cloud-era file formats move their default codec to ZSTD?
Object storage delivers only 50–100 MB/s per connection, so nearly any codec decodes faster than the data arrives, and the best default becomes the highest ratio. That is an economic condition of the cloud, not a law: on fast local disks the answer changes.
5Why is 'Parquet with ZSTD' not a single choice?
It is a pipeline of three stages: the layout (columns), lightweight encodings (dictionary, run-length, bit-packing), and then a general-purpose codec. Each stage removes the redundancy the next one is bad at. Reversing the order would destroy the type structure the lightweight encodings need.
Read moreWhy three stages
Serialization formats and sort keys
6Why does Avro need a schema registry when Protobuf does not?
Avro writes no framing bytes at all, so a reader cannot find or skip a field without the writer's exact schema. Protobuf spends about one tag byte per field, which lets a reader skip fields it does not know.
Read moreWhat the density dial controls
7Is JSON much larger than Avro?
Uncompressed, about 7–9× for a typical record. After a batch is compressed, only about 1.3–2×, because the repeated field names compress away. The real arguments against JSON are parsing CPU, no enforced contract, and ambiguous number types.
Read moreHow big are the formats, really?
8What three properties can a serialization format not all have at once?
Dense (no bytes spent describing the data), self-sufficient (readable without any external schema) and skippable (a reader can pass over fields it does not know). You get two.
Read moreThe impossibility at the centre
9Why must you never reuse a Protobuf field number?
Old data still carries the old number. If a new field reuses it with a compatible wire type, old records silently decode into the new field, with wrong meaning and no error.
Read moreThe three escapes
10Why do columnar formats prefer bit-packing to variable-length integers (varints)?
A varint's length depends on its value, so you cannot compute where the i-th value starts, and decoding takes a data-dependent branch per byte (each misprediction costs 15–20 cycles). Bit-packing uses one width per block, so offsets are computable and SIMD can decode many values at once.
Read moreVarint and zigzag
11What does zigzag encoding do, and why does it exist?
It maps signed integers to unsigned ones, 0, −1, 1, −2 → 0, 1, 2, 3, so small negative numbers stay short as varints. It exists because delta encoding produces small signed numbers.
Read moreVarint and zigzag
12How do you encode a signed integer so that comparing bytes with memcmp sorts it correctly?
Write it big-endian and flip the sign bit (XOR the first byte with 0x80). For floating-point numbers: if the sign bit is set, invert all the bits; otherwise invert only the sign bit.
Read moreOrder-preserving byte encoding
13How do you encode a variable-length string inside a composite sorted key?
End it with a terminator rather than prefixing its length, and escape the terminator byte: 0x00 becomes 0x00 0xFF, and the end marker is 0x00 0x01. A length prefix would sort strings by length first.
Read moreOrder-preserving byte encoding
14Why is UTF-16 unsuitable for sort keys, and does UTF-8 byte order give the order a human expects?
In UTF-16, surrogate pairs sort below U+E000 even though they stand for higher code points. UTF-8 byte order does equal code-point order, but code-point order is not correct for any human language. For that you need a collation sort key, which is larger and cannot be turned back into the string.
Read moreOrder-preserving byte encoding
Columnar files, Arrow and schemas
15What are the three separate wins of a columnar file format?
Projection: read only the columns needed, about 60× for 3 of 200 columns. Homogeneity: values of one type side by side encode well, typically 2–4×, even when every column is read. Skipping: per-chunk statistics let the reader avoid data it does not need.
Read moreColumnar file encodings
16How does Parquet store a nested list without losing its structure?
With two small numbers per value. The definition level says how many of the optional or repeated ancestors are present, so it records where a value is missing. The repetition level says at which ancestor a new list started. Both are bounded by the schema's depth and compress well with run-length encoding.
Read moreDefinition and repetition levels
18Why does Apache Arrow mark nulls with a validity bitmap instead of special sentinel values?
One bit per value costs only 1.56% on 64-bit integers and lets code process values without a branch, avoiding mispredictions of about 15 cycles. A sentinel steals a legitimate value from the domain and brings the branch back.
Read moreArrow and Parquet
19What does zero-copy in Arrow actually remove?
The CPU spent serializing and deserializing between components, often 60–80% of a typical Parquet-to-dataframe path through a row-oriented driver. It does not remove network transfer: the bytes still have to move.
Read moreArrow and Parquet
20What kinds of breaking change does a schema registry not catch?
Changes of meaning that keep the same structure: a unit change (grams to kilograms), a new enum member that old code does not expect, or a field that starts being null.
Read moreWriter–reader resolution
Lightweight encodings
21Why does the order of rows often matter more than the choice of codec?
Run-length encoding on a column with k distinct values produces about N runs when rows are in random order (ratio about 1), but exactly k runs when sorted (ratio N / k). With a million rows and 100 values, that is 1× against 10,000×. Sorting also narrows each chunk's min/max statistics, so it buys skipping too.
Read moreRow order dominates the codec
22When does dictionary encoding stop paying?
When the dictionary itself gets large. For a million 20-byte strings, 1,000 distinct values give about 15×, but 500,000 distinct values give only 1.6×, and the 10 MB dictionary no longer fits in cache. Parquet gives up on the dictionary for a column chunk once the dictionary page passes 1 MB.
Read moreDictionary encoding
23How can a dictionary make queries faster, not only smaller?
WHERE country = 'India' becomes one dictionary lookup to get a code, then a SIMD scan of small integers, often 10–50× faster than comparing strings. Grouping by a dense code becomes array indexing, with no hash table.
Read moreThe real win is operability
24What is special about run-length encoding for aggregation?
Aggregates run over the runs, not the rows: SUM is the sum of value × count per run. A column compressed 1,000× also aggregates about 1,000× faster, so the ratio becomes a CPU speedup.
Read moreRLE computes without decoding
25What is the difference between delta encoding and frame-of-reference encoding?
Delta stores each value minus the previous one, so decoding value i needs every value before it: no random access. Frame of reference subtracts one constant per block, so each value decodes on its own.
26How can one outlier ruin a bit-packed block, and what does patched frame of reference (PFOR) do about it?
The bit width is set by the largest value. One value of 2³¹ in a block of 128 forces 31 bits on all of them, so 0.8% of the data sets 100% of the width. PFOR chooses a width that covers most values and stores the few exceptions separately in a patch list.
Read moreThe failure casePFOR
28To record which values are present, should you use a bitmap or a list of positions?
A bitmap costs N bits for N rows; a position list costs m · log₂ N bits for m present values. They break even at a density of 1 / log₂ N, about 5% for a million rows. Above that use a bitmap; below it, positions.
29Why do floating-point columns compress badly, and what does ALP do about it?
The low bits of the mantissa look random, so a double carries 30–50 bits of entropy and general codecs reach only 1.5–2×. ALP notices that many doubles were decimals in disguise: it finds exponents that turn them into exact integers, then applies frame of reference and bit-packing.
30Can you update a value inside a run-length run or a bit-packed block?
Not without rewriting the block. Every lightweight encoding assumes the data is appended and never changed in place, which is why transactional row stores that update in place use almost none of them.
Read moreWhat lightweight encoding forbids
General-purpose compression
31Why does Huffman coding do worst on highly skewed data?
It must spend a whole number of bits per symbol. A symbol with probability 0.9 deserves 0.152 bits but costs 1. A binary source with p = 0.99 has an entropy of 0.08 bits per symbol, yet costs 1 bit: 12× too much.
Read moreHuffman coding
32Why did ANS (and its FSE implementation) change compression?
It reaches the compression of arithmetic coding with a branch-free, table-driven state machine running at 1–2 GB/s, and interleaving several streams lets one thread decode in parallel. That, with a better match finder, is why ZSTD replaced older codecs.
Read moreANS and FSE
33Why is decompression so much faster than compression, and does the compression level affect it?
Compression has to search for repeated matches; decompression only copies bytes from offsets it is told. The level controls only how hard the encoder searches, so decompression speed hardly changes with level.
34Should we still use gzip?
Only for compatibility. ZSTD at level 3 gives about the same ratio as zlib at level 6 (about 3.1), with 5–8× faster compression and 3–4× faster decompression.
Read moreThe LZ family
35Why do small messages compress badly, and how do you fix it?
The compressor starts with an empty window, so there is nothing to match against, and its tables can be larger than the message. A trained dictionary, about 110 KB built from sample records, pre-fills the window and tables and gives 2–5× on small records. The price is a permanent, versioned artifact that every reader needs.
Read moreTrained dictionaries
36What does the compression block size control?
Three things at once. The ratio, which rises up to about 64–256 KB and then flattens. Read amplification: reading a 100-byte row from a 1 MB block decompresses about 10,000× more than needed. And how much data one corrupted bit destroys. It also limits how much decoding can run in parallel.
Read moreBlock size is three decisions
37Should checksums be turned off for speed?
No. CRC32C has a dedicated CPU instruction and runs at 10–20 GB/s. Compressed data turns corruption into plausible-looking bytes, so the checksum is the only thing that will notice.
38What is content-defined chunking?
Cutting data into chunks at boundaries chosen by a rolling hash of the content itself, so inserting bytes moves only one nearby boundary. Each distinct chunk is stored once, which finds redundancy across files and snapshots, far beyond any compressor's window.