What do I do, and what does each option cost?

Choose a transport

Records must move from a producer to consumers: a synchronous call, a queue, publish/subscribe or a log? Three questions pick the transport; five settings then make it safe.

Handle overload

Work arrives faster than it can be served: break the circuit to a failing dependency, buffer a burst, slow the source, refuse at the door, or shed? Four questions pick the response; two checks keep it from spreading.

Choose a partitioning

Records are split across N partitions or shards: round-robin, hash or range, by which key, with how many partitions, and what happens to skew, order and failures?

Make effects repeatable

Messages can arrive more than once and consumers can crash mid-way: list every effect, make each one safe to repeat, commit progress after the work, and prove it by crashing the consumer.

Choose a layout

Data is about to be stored: row, columnar, PAX (columns within row groups) or key-value? Six questions, from the storage medium to the share of columns each query touches.

Choose a mutation strategy

Stored data changes: update in place, rewrite whole files (copy-on-write), or write changes separately and merge on read with delete vectors or equality deletes? Four questions, then the compaction interval and the metric that predicts failure.

Choose a clustering strategy

Rows can be physically ordered so queries skip most files: a single sort key, Z-order across a few dimensions, bucketing for joins, or no clustering at all? Two questions, then the checks that decide whether it pays its rent.

Measure your medium

Before choosing any block, page or file size: measure your storage's latency, bandwidth and achievable concurrency, then derive the minimum efficient request and the queue depth your reads need.

Design backups you can restore from

A backup plan for a database or a data lake: name what it defends against, set the data-loss window by log shipping, set the restore time by base age and restore bandwidth, decouple the backup in time and authority, and test it by restoring.

Choose a lookup structure

You need to find records by key: a static structure, a hash table, a skip list, a B-tree or an LSM-tree? Four questions, each removing more of the design space, then a check of the failure mode.

Choose an ordering structure

A query needs data in order: a heap, an existing order, an in-memory sort, an external sort or a distributed sort? Four questions, starting with how much order you really need.

Choose a sketch

An exact answer is too big to keep: which sketch answers your question (membership, distinct count, frequency, quantile, similarity or a sample), and will it survive merging, duplicates and the questions asked later?

Choose a bit-level structure

A structure is read-mostly and could be packed far smaller: is the saving worth it, and should it be a bitmap, an Elias-Fano list or a packed array with rank/select? Plus when min/max statistics can prune at all.

Choose a serialization format

Data is written by one system and read by others, possibly years later: where does the schema live (compiled in, in a file header, per field, or in a registry), and is the data laid out by row, as a columnar file, or as columnar memory?

Choose lightweight encodings

Choosing how each column of a columnar file is encoded: decide first whether it must stay queryable, then row order, then dictionary, run-length, delta, frame-of-reference and bit-packing by what the values look like.

Choose general-purpose compression

Whether to put a general-purpose codec (ZSTD, LZ4) on top of your data at all, and if so with what block size, level, dictionary, checksum and order relative to encryption.

Design an index

Designing the indexes for a table from its queries: spend the one clustered order first, skip indexes past the selectivity crossover, then choose bitmap, covering, partial and composite indexes, and drop the unused ones.

Choose an index family

A query filters by some key: does the physical order already answer it, and if not, does the predicate's shape call for a hash, an ordered index, an inverted index, a spatial index or approximate nearest-neighbour search?

Choose a concurrency scheme

Concurrent transactions write to the same data: locking, optimistic, commutative operations, partitioning or deterministic execution? Measure the conflict rate, shorten transactions, then follow five questions to one scheme.

Design a commit

Building or choosing the commit path of a store: name the one atomic write it rests on, derive the commit-rate ceiling, and decide what an abort and a retry will cost.

Choose a distributed commit

One operation writes to several systems: make it all-or-nothing, or let each system commit on its own and handle failures another way? Six questions lead to one of six designs, each with its costs and checks.

Choose an isolation level

Transactions on one database read and write the same rows: rewrite the operation, push the rule into a constraint, or pick read committed, snapshot isolation or serializable? Four questions, cheapest answer first.

Decide on consistency

Data is replicated across machines or regions: does your rule need coordination at all, can you split the budget so replicas act alone, or do you need real-time ordering? Five questions, cheapest answer first.

Check operators for cost and correctness

Reviewing a query plan: find where the row count peaks, which operators block, what each holds in memory and which orderings it needs; then a separate pass for seven silent wrong answers.

Choose a join algorithm

Two inputs must be joined: a non-equality join, an index lookup, a merge, a local join on existing partitions, a broadcast or a shuffle, and then which hash variant fits memory? Then check key frequency and NULLs, which override it all.

Choose an aggregation strategy

A query groups and aggregates: does the function allow partial results, is an exact answer needed, and does pre-aggregating before the shuffle pay? Three questions, then sizing, sort reuse and determinism checks.

Plan a sort

A query needs data in order: check that order (not just grouping) is needed, take it for free if the data already has it, then size the sort: passes, fan-in, key normalization, and sampling for a distributed sort.

Choose a dedup strategy

Duplicates must be removed or counted: a sketch for counting, streaming dedup over sorted input, hash dedup in memory, or sort-based dedup when it will not fit? Plus wide keys and why approximate dedup can lose rows.

Diagnose a bad plan

A query got a bad execution plan: check estimated against actual rows first, then statistics, correlation, subqueries, the number of joined tables and pushdown, and only then the cost settings.

Find why a query reads so much

A query reads far more data than its result needs: check that the filter reaches the scan on a clustered column, that it is not negative, that runtime filters, projection and Top-N apply, before making execution faster.

Speed up a reasonable plan

A query has a sensible plan and reads only what it needs, but is still too slow or too frequent: reuse the answer, reuse the plan, adapt at runtime, compile, or cache a shared subplan, in order of payoff.

Choose an execution model

A workload must run: on one machine, or across many as a pipelined or materialized dataflow, bulk-synchronous rounds, actors or serverless functions? Three questions, then the arithmetic that decides how failures are survived.

Plan the parallelism

Before adding workers: find the ceiling set by the largest partition, the stages set by pipeline breakers, whether the costly operator can be split at all, how to size tasks, and when speculation and elastic scaling pay.

Handle skew and elasticity

A stage's last tasks run far longer than the rest: plot duration against input bytes to tell imbalance, a straggler and skew apart, then apply the fix that works for that one.

Design a shuffle

Data must be repartitioned across machines: compute bytes per connection against the storage floor, cap the width, shrink the data first, then choose hash or sort, a shuffle service, compression and serialization.

Choose who owns the memory

An engine allocates memory for data it processes: leave it to the garbage collector, or take it off-heap as an arena, a typed pool, a dictionary or capped buffers? Four questions, then two checks on placement and peak.

Tune a buffer pool

A database caches disk pages in a buffer pool: isolate scans, use Clock, or pay for 2Q or ARC? Three questions pick the replacement policy; three checks find the pool's usable size and the failures upstream of the policy.

Ration memory across queries

Many queries share one machine's memory: list what cannot spill, check where grants come from, size grants in steps, decide between reservation and over-commit, and keep the pressure loop from oscillating.

Plan for running out of memory

When a query outgrows its memory, it spills to disk or is killed: list what cannot spill, compute what spilling really costs on your hardware, guard against one oversized key, and make sure a killed query retries with more memory.

Size a cache

Before buying cache capacity: measure reuse, find out whether the cache must grow with the data, size to just past a knee, check each tier pays, and check the percentile you promised.

Choose what to cache

Something is slow and a cache could help: cache file blocks, metadata, final results, a materialized view, overlapping fragments, or the absence of a key? Two checks and five questions lead to one of six choices.

Keep a cache honest

A cache sits in front of data that changes: decide how writes reach it, how stale an entry may get, and how to stop expiry and restarts from flooding the store behind it.

Plan a partition move

Adding or removing nodes from a partitioned store: move whole logical shards instead of changing their count, size the move by its throttle, check free space for the peak, run copy → catch up → cut over → retire, fence clients with the map's version, and keep the move away from automatic failure detection.

Choose a recovery mechanism

A step can fail part-way and must be recovered: compensate, keep a standby, recompute from lineage, restore a checkpoint, or both? Four questions decided by reversibility, the recovery-time target and whether the result can be recomputed.

Make a repeat harmless

Recovery will redo work, so an effect may land twice: use an operation that absorbs repeats, store progress with the effect, commit through the sink, or deduplicate at the receiver? Four questions, cheapest first.

Review a design for runaway failure

Seven questions for a design review that find the failures a system causes itself: feedback loops, degraded-mode amplification, shared dependencies, blind health checks, head-of-line blocking, deferred work and unpriced recovery.

Recover from a stuck overload

The trigger is gone but the system stays down: why shedding a little load and restarting both fail, how much load must really go, and the five-step procedure that gets out.

Share a cluster or split it

Several teams need compute: one shared pool, a shared pool with guaranteed floors, or separate clusters? Three questions about correlation, guarantees and how fast capacity comes back, and the price of each answer.

Cut the bill

A data platform costs too much: find where the spend concentrates, then try the escapes in order of price (change the workload, the meter, the layout, the architecture), knowing what each one costs to use.

Handle late data for each consumer

Records arrive after the watermark: drop them (and count them), divert them, or re-fire the window as totals or as deltas? Three questions per consumer, decided by what a missing record costs it, how much state you can keep, and what its sink can absorb.

Find the backpressure culprit

A streaming job is lagging and half the graph reports backpressure: read it from the sink backwards, find the first operator that is not blocked, then check for a hot key, a stalling state backend, checkpoints slowed by the backlog and missing headroom.

Choose how to capture changes

Rebuild, compare partitions, diff snapshots, announce from the writer, read the log, install triggers or poll? Six questions, cheapest first: each rung down costs the source less and misses a nameable class of change.

Set up maintenance for a table

A lakehouse table under continuous writes: stop small files at the source, pick a target size, trigger compaction by cost, set retention from the longest reader, rewrite deletes per file, clean orphans safely, verify every rewrite and tier only what has been dormant long enough.

Model a dimension and its history

A new dimension, such as customer or product: declare the grain it serves, reuse a conformed dimension, choose the history type per attribute, key the versions and load late facts against the right one.

Ship a schema change without breaking readers

Before changing a schema that others read: test the operation, know your identity rule, pick the compatibility mode from the deployment order, run renames as expand/contract, and check the changes no registry sees.

Decide what a failed check does

A data check failed: warn, quarantine the bad rows, block a partition, block everything, or restate what was already published? Four questions, set per check by the cost of bad data against the cost of no data.

Retire a table safely

Deleting tables nobody seems to use: narrow the candidates with lineage, rank them by real reads, assume a long quiet period is not proof, and make hidden use visible through a denial channel before anything is deleted.

Choose a coordination model

Steps of a pipeline or process must run in order: one coordinator or chained events, and if a coordinator, a graph of tasks or durable execution? Two questions decided by ownership boundaries and by where the state between steps lives.

Decide what may page you

Set up alerting for a data platform: page only on symptoms consumers feel, measure freshness from the data, write objectives per opportunity, alert on burn rate, suppress by lineage, apply the severity-to-action test, and watch the watcher from outside.

Protect a sensitive field

A column holds personal or secret values: mask it, hash it with a key, generalize it, add noise to aggregates, tokenize it or encrypt it? Five questions, each option preserves a different structure, and what it preserves is what it leaks.

Decide whether to specialize

A new workload (search, vectors, graphs, geospatial, time series) seems to need its own engine: stay in the general database, use an extension, or adopt a specialized system, and what the second system really costs.

Design an interface between systems

Data must cross from one system or team to another, as files, API calls or an ongoing sync: decide where the schema lives, what happens when a transfer dies mid-way, what position a sync resumes from, and write down the agreement both sides depend on.

Plan a benchmark

Before running a benchmark that will decide something: predict the result, register the decision rule, pin and record the environment, generate load that cannot hide overload, measure the noise floor, fix the run count and order, and keep it rerunnable.