Playbooks
What do I do, and what does each option cost?
- 1
Choose a layout
Data is about to be stored: row, columnar, PAX (columns within row groups) or key-value? Six questions, from the storage medium to the share of columns each query touches.
Choose - 2
Choose a mutation strategy
Stored data changes: update in place, rewrite whole files (copy-on-write), or write changes separately and merge on read with delete vectors or equality deletes? Four questions, then the compaction interval and the metric that predicts failure.
Choose - 3
Choose a clustering strategy
Rows can be physically ordered so queries skip most files: a single sort key, Z-order across a few dimensions, bucketing for joins, or no clustering at all? Two questions, then the checks that decide whether it pays its rent.
Choose - 4
Measure your medium
Before choosing any block, page or file size: measure your storage's latency, bandwidth and achievable concurrency, then derive the minimum efficient request and the queue depth your reads need.
Design