Scenarios

Readers are in many teams, so binding to code at compile time is out. Teams upgrade at different times, so consumers must tolerate unknown fields: either Protobuf-style tags (about a byte per field, field numbers fixed forever) or Avro with a registry (no framing bytes, but the registry becomes a dependency). Records are individual messages, so reference the schema by a 4–5-byte fingerprint rather than embedding it.

Every schema ever registered must resolve forever: replicate the registry, back it up as durably as the topic, and keep a copy of every schema in the audit store.

Transitive: each new schema must be compatible with all earlier versions, because replays start at offset zero and a failed replay lands on another team months later.

Units in field names; a change of meaning means a new field; never reuse a field number; contract tests with expected value ranges.

Protobuf costs about 10 bytes per 10-field record and a permanent tag namespace; Avro saves about 6 bytes but makes the registry a hard dependency. After ZSTD on batches, all formats, even JSON, are within 1.3–2× in size, so decide on contract and CPU, not size.

Use the break-even D / B > r / (r − 1). From S3, B is about 50–100 MB/s per connection and ZSTD decodes at about 1 GB/s, so D / B is 10–20, far above the threshold of 2 or less: a higher ratio helps reads. Decoding speed barely changes with level, so reads will not slow down.

That is where the cost lands: level 19 compresses 50–100× more slowly. If the nightly writer needs 10 core-hours at level 3, level 19 may need hundreds, so check it fits the batch window; compress files in parallel, or use a middle level (9–12) if the writer is on a critical path.

Gains flatten: going from a ratio of 4 to 5 saves 20% of the bytes. Measure on real data.

High levels for cold object-store data written once; low levels for hot data or local copies. On machines with local NVMe caches, B is 3–7 GB/s and ZSTD can lose, so keep a fast codec there.

The aggregation reads 2 of about 10 columns and gains from projection and encoding. A point lookup needs whole rows: it touches every column chunk, decompresses each page, and on object storage waits two round trips per file for the footer (about 2 × 30–70 ms). The file is organized by column chunk; the query wants a row.

Keep Parquet for scans, and add a path for lookups: a row-oriented or key-sorted store, such as an LSM store keyed by parcel ID with an order-preserving key encoding. Or sort and partition the lake by parcel so statistics let the query skip most chunks, and cache footers.

A second copy of the data, and a pipeline to keep it in sync. Do not try to tune Parquet into serving point lookups; the gap is structural.

The engine compares keys as raw bytes with memcmp, so every part must encode to bytes that sort in the intended order.

Variable-length, so escape 0x00 as 0x00 0xFF and end with 0x00 0x01; then ("ab", "c") and ("a", "bc") stay distinct and correctly ordered. If tenants need language-correct order, store a collation sort key next to the original (larger, and not reversible).

Big-endian, with the sign bit flipped (XOR the first byte with 0x80).

Apply the float rule (sign bit set: invert all bits; otherwise flip only the sign bit), then invert all eight bytes for descending order. Choose and document where NaN and negative zero sort. If nulls are allowed, add a type tag or reserved byte for each part.

Escaping expands keys, keys cannot be decoded in place, and sort keys take extra storage. Test with a property: random tuples sorted logically must equal the same tuples sorted by their encoded bytes.

Most likely a change of meaning, such as dollars to cents, or a new unit or enum value. Schema compatibility checks only structure, so the change passed. Compare value distributions before and after the deploy, find the first bad offset, and look in the producer's diff for unit or rounding logic.

Once the field is fixed, quarantine or replay the affected range.

Units in field names; "new meaning, new field"; contract tests on value ranges and null rates at the pipeline's edge; canary consumers; and failing fast on out-of-range values (quarantine the record and alert) instead of silently passing them through.

This class of failure never disappears; validation only narrows it.

Parquet minimizes bytes at rest; Arrow minimizes CPU cycles in use. In memory, Arrow is commonly 2–10× larger (no bit-packing, fixed-width values, validity bitmaps, 64-byte alignment), so a 5 GB Parquet file may need 10–50 GB of RAM.

Reading Parquet into Arrow still means decoding: decompressing, expanding dictionaries, run-lengths and bit-packing, and building bitmaps. That is real CPU; measure it.

It removes serialization between Arrow-aware components, for example between processes or over Arrow Flight. Bytes still cross the network. The large win is replacing a row-oriented database driver, where encoding and decoding take 60–80% of the time.

Read only the needed columns, stream one row group at a time to bound memory, and benchmark the whole conversion before promising anything.

262,144 ÷ 100 ≈ 2,600× more bytes decompressed than needed, about 0.26 ms of decoding at 1 GB/s before the disk read even counts.

Move to 4–16 KB blocks: 40–160× amplification and 4–16 µs of decoding. The ratio will fall (smaller window, more table overhead), so add a trained dictionary of about 110 KB to win back 2–5× on small records.

The dictionary is a permanent, versioned artifact: keep every old one and store its ID in each block header. Add a CRC32C per block, so a flipped bit costs 4–16 KB instead of 256 KB. Scans over this store get a little slower, and compression CPU per byte rises.

Keep a second copy in large-block Parquet for them.

Delta-of-delta: zero for a regular series, so most timestamps cost about 1 bit (Gorilla reports 96% at one bit).

First test whether they are decimals in disguise; if so, ALP turns them into integers and then frame of reference and bit-packing, at several GB/s. If they are truly smooth floats, use Gorilla's XOR encoding, but expect poor results on signals that oscillate around zero.

Delta and XOR form serial chains, so restart every 64–128 samples to allow point reads at block granularity. The restart interval trades ratio for point-read cost.

Sort or cluster by series, then time, so each series is contiguous.

Gorilla's 12× figure is for regularly sampled metrics; do not assume it for irregular data. And the design assumes append-only blocks, so late samples need a separate buffer and compaction.

Row order. The streaming job writes in arrival order: categorical columns have runs of length about 1, deltas are random, and per-file min/max ranges overlap, so skipping fails too. The batch job sorts, so run-length encoding gets k runs instead of N, deltas are small, and file ranges are disjoint.

Add a compaction step that re-sorts files by the column most queries filter on, low-cardinality columns first. It improves ratio and skipping, not just file count, so justify its budget with both.

A file has only one sort order. If queries filter on several columns unpredictably, consider Z-order or Hilbert ordering, which gives each column partial locality at the cost of the leading column's strong locality.

Compression fails by degrading, not by erroring. Compare the compression ratio per column per day. Typical causes: a new customer with high-cardinality IDs where there used to be a few values (the dictionary passed 1 MB and fell back to plain encoding), a writer that stopped sorting, a changed batch size, or a column that started holding random floats. Check dictionary-fallback counters and run lengths per column.

Alert on per-column ratio as a data-quality signal.

Ingestion is measured on throughput and the query team on latency, so nobody owns the ratio. Give one team the budget for both the CPU and the bytes.

After good lightweight encoding, a general-purpose pass adds only 1.2–1.5×. With r = 1.35, the break-even needs D / B > 1.35 / 0.35 ≈ 3.86, so decoding at 3.86 × 5 GB/s ≈ 19 GB/s. One ZSTD core decodes about 1–1.5 GB/s; you would need more than ten cores doing nothing else. Reads would get slower.

On object storage at 100 MB/s per connection, the same codec is easy: the threshold is 386 MB/s and ZSTD gives 1 GB/s.

No general-purpose pass on the NVMe copy (or LZ4 at 3–4 GB/s if some saving is needed), and ZSTD on the object-store copy. The trade-off is a larger NVMe copy.

Level 3 compresses at about 250 MB/s per core with a ratio of about 3.1; level 19 at 5–10 MB/s (say 7) with a ratio of about 3.6.

Level 3: 2 × 10¹² ÷ 250 × 10⁶ = 8,000 core-seconds, about 2.2 core-hours a day. Level 19: 2 × 10¹² ÷ 7 × 10⁶ ≈ 285,700 core-seconds, about 79 core-hours, or 3.3 cores running all day.

645 GB against 556 GB stored per day, a saving of about 89 GB a day.

Ingestion cannot afford level 19 at line rate, and you do not yet know which data will be read often. Write at level 3, then rewrite at 9–19 during nightly compaction for data that is read many times; reading costs nothing extra. Past a ratio of about 4, more ratio helps storage cost much more than query time.

The team paying for the CPU and the team gaining the bandwidth are different, so set the policy centrally.

Compress first, then encrypt: ciphertext is random and does not compress.

Per-column encryption keeps per-column compression possible, but the footer exposes each column's compressed size, which reveals something about its distribution, and ciphertext length leaks content length (the CRIME and BREACH class of attack). For data at rest this is usually acceptable; record it as a known residual risk.

Keep a checksum per compressed block, or use authenticated encryption: one flipped bit in a compressed block destroys the whole block.