Playbooks
What do I do, and what does each option cost?
- 1
Choose a serialization format
Data is written by one system and read by others, possibly years later: where does the schema live (compiled in, in a file header, per field, or in a registry), and is the data laid out by row, as a columnar file, or as columnar memory?
Choose - 2
Choose lightweight encodings
Choosing how each column of a columnar file is encoded: decide first whether it must stay queryable, then row order, then dictionary, run-length, delta, frame-of-reference and bit-packing by what the values look like.
Design - 3
Choose general-purpose compression
Whether to put a general-purpose codec (ZSTD, LZ4) on top of your data at all, and if so with what block size, level, dictionary, checksum and order relative to encryption.
Design