Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

What general-purpose compression forbids

What does general-purpose compression forbid, and what does each escape cost?

The standing questions, and an organizational constraint

What does this system assume that it cannot verify? That the distribution of the data still looks like the data used to choose the level, the block size and the dictionary. Nothing checks this. When a new customer arrives with UUIDs where there used to be a few enum values, the ratio silently halves, and the result shows up months later as storage growth and slower queries, never as an error. , and that is the kind of failure that organizations are worst at detecting. You should alert on per column as a data-quality signal. Almost nobody does.

Who pays when the assumption breaks? Here the constraint is not physical or informational. It is organizational. The producer pays the CPU for compression. The consumer receives the benefit in and . In most companies these are different teams with different budgets. The ingest team is measured on throughput and cost per ingested GB, so it picks level 1 and moves on. The query team is measured on , and complains about scan times. The setting that would improve the whole system makes one team's number worse. So the technically correct level is systematically under-chosen, and no benchmark fixes this, because benchmarking was never the bottleneck.

Cores can be bought, and turns bought cores into fewer bytes at a computable rate, but the cores and the bytes come from different budgets.

This has the same shape as schema contracts: the producer bears the cost of keeping the contract, and the consumer receives the benefit. That is why schemas rot most in organizations where the producing team is busiest.

You cannot have compression and addressability together. Once the bytes are entropy-coded, there is no value i; there is only a block. Everything that comes after (block sizing, restart frames, independent chunks) is damage control for a property you gave up on purpose. The same number governs how much redundancy you find, how much you decode to read one row, how much you lose to one bad bit, and how much parallelism you get.

The escapes and their prices

  • Smaller blocks buy back access and limit the . They cost ratio and table overhead.
  • Independent frames buy back parallelism. They cost ratio at every boundary.
  • Trained dictionaries buy back ratio on small inputs. They cost a permanent coordination dependency.
  • Skipping the pass entirely buys back CPU. It is correct more often than people think, when the pipe is fast.