Choose lightweight encodings
A columnar file stores each column separately, and each column can be encoded to suit its values: replace repeated strings with small codes, store runs once, store small differences instead of large numbers. These encodings are cheap to decode, and some can be queried without decoding at all. How should each column be encoded?
You will end with an encoding per column, chosen from what its values look like and how it will be queried.
Work through these per column, in order.
Work through it
First: will this column be filtered, joined or grouped?
This overrides compression ratio. If yes: prefer encodings the engine can query directly. A dictionary at 8× that lets filters run on the codes beats a scheme at 15× that must be fully decoded first. Decide this before comparing sizes.
If you can choose the row order, start there
Sorting rows by the right columns creates long runs and small differences, which every encoding below exploits. It dominates everything else on this list. Sort order is scarce, though: data can be sorted by only one key.
Few distinct values: dictionary
If the number of distinct values is below about a tenth of the rows and the values are wide: use a dictionary (store each value once, and a small code per row). If distinct values are few and the data is sorted: dictionary, then run-length encoding (store each run of repeats once).
Values that change gradually: delta or frame-of-reference
If each value is close to the previous one (timestamps, sequence numbers): store differences (delta) if you will only scan; store each value relative to a per-block base (frame-of-reference) if you need random access. Prefer frame-of-reference unless the overall range is large compared with each block's range.
A narrow range in a wide type: frame-of-reference plus bit-packing
If a 64-bit column only holds values from 0 to 1,000: subtract the block's minimum and pack each value into the bits it needs. Always.
Look at the bit-width histogram, not the maximum
The bits needed are set by the largest value in the block. If the 99th percentile is far below the maximum: store the few large values separately as exceptions instead of widening every value for them. A fraction of a percent of values can otherwise set the whole cost.
Sorted strings: front coding
Store each string as the length of the prefix it shares with the previous one plus the rest, restarting with a full string about every 16 entries so lookups can jump in.
Floats: check for decimals in disguise first
Many floats are prices or measurements with a fixed number of decimal places. If so: encode them as scaled integers. Only otherwise reach for a float-specific codec such as XOR against the previous value.
Then decide whether general-purpose compression on top is worth it
After good lightweight encoding, a general-purpose codec usually adds only 1.2–1.5× more. See Choose general-purpose compression.