What's going wrong?
Every symptom from the How it works cases, each linked to the page that explains it.
Physical Substrate & Performance Laws
A nightly job over thousands of small files takes over an hour
Storage media: what each one is made of and what it forces
Pipeline is slowA drive rated at a million IOPS delivers a small fraction
Storage media: what each one is made of and what it forces
Cluster is slowA scan over data already in RAM is still slow
CPU and memory: the machine is a guessing machine
Query is slowPer-thread counters still make added threads slower
CPU and memory: the machine is a guessing machine
Service is slowA fix that didn't workA filter loop is several times slower on unsorted data
CPU and memory: the machine is a guessing machine
Query is slowWith many servers in a fan-out, p99 jumps while the network is quiet
The network: a shared medium with invisible neighbors
Service is slowA fix that didn't workAdding servers makes throughput go down
Amdahl, Gustafson and the Universal Scalability Law
Service is slowA fix that didn't workAt high utilization a modest traffic rise sends p99 to seconds
Little's Law and the queueing wall
Service is slow
Data Movement
Every checkout fails while an optional service is down
What changed: from one machine to two
Service failsA customer is charged twice after a call times out and is retried
What changed: from one machine to two
Service has duplicatesLast month's messages are gone when you try to re-read them
Queue versus log: where progress state lives
Stream is missing dataCustomers get two confirmation emails after a consumer restart
Delivery semantics: the basics
Stream has duplicatesOne bad record stops a partition while the service shows healthy
Eight transports on three switches
Stream is stuckEverything said it was fineA consumer's in-memory buffer grows until the process is killed
Buffering · Backpressure
Stream runs outRetrying at once against a rate limit makes the partner slower
Rate limiting, throttling and pacing
Service is overloadedTraffic arrives at several times the usual rate
Load shedding
Service is overloadedEvery checkout waits on a slow partner check
Rate limiting, throttling and pacing
Service is slowOne partition falls hours behind; added consumers sit idle
Two kinds of skew: different problems, different fixes · Partitioning: the four strategies
Stream has a hot spotA fix that didn't workEvents for one order are processed out of order
Message keys and key-based ordering · Dead-letter queues
Stream is wrongThe network bill is high for many small events between regions
Compression on the wire
Bill costs too muchThe first requests after a quiet night fail or take seconds
Connection pooling, keep-alive and heartbeats
Service failsMost producer CPU goes to compression and TLS
Which network choices change behavior
Stream is slowA connection shows open after a reboot but carries no data
Connection pooling, keep-alive and heartbeats
Service is stuckEverything said it was fineReplaying to fix a bug doubles the replayed days' revenue
Replay
Dashboard has duplicatesA fix that didn't workThe warehouse holds slightly fewer rows than produced, with no error
Composition across a chain, and verification
Dashboard is missing dataEverything said it was fineDays the fix never touched change by a few cents after a replay
Replay
Dashboard is wrongA fix that didn't work
Storage: Physical Organization
A lookup instant on a laptop takes seconds on object storage
Data at rest and the one ratio
Query is slowThe bill's request line is larger than its storage line
Data at rest and the one ratio · Object storage and the lakehouse
Bill costs too muchA report slows after upstream JSON keys are renamed, with no error
Opaque values, documents and where order comes from
Query is slowA one-row lookup gets slower after moving to columnar files
Layout models: row, column and PAX
Query is slowA fix that didn't workA nightly job correcting a few rows takes hours and grows the bill
Mutation strategies: what happens when a value changes
Pipeline is slowReports get a little slower every week after switching to delta files
Mutation strategies: what happens when a value changes
Query is slowA fix that didn't workAfter a backfill, a query that skipped most files scans everything
Partitions, buckets, row groups and zone maps
Query is slowA lookup for a dozen rows opens every file in the table
Physical clustering: which units can we skip?
Query is slowAdding parallel readers makes a disk scan slower
The cost of one access
Query is slowA fix that didn't workObject storage reads are fast in parallel and awful one after another
The cost of one access
Query is slow
Persistence & Durability
Confirmed writes are missing after a power cut, with no error logged
The frame: what "durable" means
Database is missing dataEverything said it was fineA file written by temp-and-rename is empty after a power cut
Journaling: the filesystem's own WAL
Service is missing dataForcing writes to disk makes each commit take tens of milliseconds
Write-ahead logging (redo)
Database is slowA fix that didn't workA retried fsync reports success, but the data is gone
Fsync, flush barriers and O_DSYNC
Database is missing dataEverything said it was fineCommits stop at a fixed rate however many clients are added
Group commit
Database is slowCrash recovery takes far longer than its target
Checkpointing: sharp, fuzzy, incremental
Database is slowThe database stops when old log files fill the disk
Log truncation and retention policies
Database runs outA mistaken DROP replicates instantly and the last backup is hours old
Backup and point-in-time recovery
Database is missing dataOne fsync takes milliseconds on a laptop and microseconds on a server
Battery-backed memory and NVRAM
Database is slowCommit throughput collapses overnight while health checks stay green
Battery-backed memory and NVRAM
Database is slowEverything said it was fineCorruption is found after the last good backup has expired
End-to-end checksums, scrubbing and bit-rot detection
Table is missing dataAfter a clean recovery, a value matches neither old nor new
Torn pages and double-write buffers
Database is wrongEverything said it was fineA promoted replica is missing the last confirmed writes
Durability across machines: replication, quorums and erasure coding
Database is missing dataA confirmed transfer never reaches any downstream system
Durability across a pipeline
Stream is missing dataA consumer off for days loses the start of its backlog
Durability across a pipeline
Stream is missing dataA day is marked done but never archived
Durability across a pipeline
Pipeline is missing dataEverything said it was fine
Core Data Structures
Two structures with the same big-O are several times apart
The big picture: a data structure is a bet
Database is slowDoubling the data makes a hash aggregation ten times slower
Hash tables
Pipeline is slowAn index with random keys outgrows its data and wears the SSD
B-trees and B+trees · LSM-trees
Database runs outA join on validity ranges runs for hours as a cross product
Interval trees and stabbing queries
Query is slowA small top-N query runs for minutes and writes huge temporary files
Heaps and priority queues
Query is slowOne sort task gets ten times the data and runs for hours
Sorted runs and merge structures
Cluster has a hot spotA sliding-window maximum runs out of memory after hours
Monotonic queues and sliding windows
Stream runs outLookups for keys that don't exist cost as much as real ones
Bloom filters · Cuckoo, XOR and ribbon filters
Database is slowCounting distinct IDs needs tens of gigabytes of memory
HyperLogLog, Theta and KMV
Pipeline runs outA bitmap per value would need terabytes for a many-valued column
Bitmap indexes and Roaring bitmaps
Table runs outFile statistics exist, yet a prefix filter opens every file
Zone maps and min/max indexes
Query is slowA reader thread crashes about once a day
Epoch-based reclamation and hazard pointers
Service failsMore threads make the ingest rate drop while the CPU is full
The currency is coordination · Lock-free queues: SPSC and MPSC ring buffers · Concurrent hash maps
Service is slowA fix that didn't work
Encoding, Serialization & Compression
Stronger compression makes scans slower on fast local disks
Bytes become an output
Query is slowA fix that didn't workA unit change shrinks values a thousandfold and every check passes
Columnar files, Arrow and schema resolution
Dashboard is wrongEverything said it was fineReplaying a topic from the start crashes on old records
Serialization: where the description lives
Stream failsNegative numbers sort after positive ones in a key-value store
Serialization: where the description lives
Database is wrongA column with few distinct values compresses badly
Dictionary and run-length encoding
Table costs too muchThe same data is several times larger when written by streaming
Row order dominates the codec
Table costs too muchOne flipped bit makes a whole block of records unreadable
Block size, level and trained dictionaries
Database is missing dataEach small lookup reads and decompresses a whole large block
General-purpose compression: the last pass
Database is slow
Indexing
A new index is ignored and the query still scans the table
Why indexes exist and when they win
Query is slowA fix that didn't workExtra indexes leave reads slow and make inserts much slower
Why indexes exist and when they win
Table is slowA fix that didn't workInserts with random UUID keys are slow and the table is huge
Primary versus secondary: identity is not location
Database is slowA covering index quietly becomes slow again on a busy table
Covering indexes: the index becomes a narrow copy of the table
Query is slowA composite index is used but still reads a large range
Composite indexes: everything follows from lexicographic order
Query is slowAn index works in the console but not from the application
Partial indexes: the one place skew is an asset
Query is slowA search for a word inside text takes minutes
Set membership: inverted indexes
Query is slowAfter a late batch of old events, recent-time queries read far more
Time: the nearly redundant index
Query is slowA geo index exists but a radius search reads much of the table
Two and three dimensions: the first real loss
Query is slowExact nearest-neighbour search over many vectors is far too slow
High dimension: exactness becomes unaffordable
Query is slowA record just created isn't found by search for a few seconds
The consistency boundary, local and global indexes
Service is staleDeleted records keep showing up in search for days
Indexes under MVCC, and maintenance
Service is staleSearching by a non-shard key has to ask every shard
When the map gets built: paying later, or paying less
Database is slow
Transactions & Concurrency Control
Serialization failures climb far faster than the order rate
Why concurrency is a different kind of problem
Database failsp99 jumps to seconds while CPU and disks sit idle
Why concurrency is a different kind of problem
Database is slowThe same count run twice in one transaction returns different numbers
Locks and latches: the distinction that has to come first
Database is wrongDropping an unused index makes a transaction lock the whole table
Lock granularity and intention locks
Database is stuckDeadlocks appear when two users pay each other at once
One graph, kept free of cycles
Database failsSwitching to retry-on-conflict pins the CPU and completes fewer orders
Optimistic concurrency control (OCC)
Database is slowA fix that didn't workTwo people go off shift at once and nobody is left on shift
Serializable snapshot isolation (SSI)
Database is wrongEverything said it was fineOne counter row that every order updates becomes a hot row
Commutativity: removing the conflict
Database has a hot spotA long report makes a table triple in size
Multiversion concurrency control (MVCC)
Database runs outA long report fails with "snapshot too old"
Multiversion concurrency control (MVCC)
Query failsA commit times out and nobody knows if it happened
Begin, commit and abort
Database is unclearA nightly job fails on a commit conflict and loses its work
Optimistic commit on a metadata log · Compare-and-swap: the object store's only transaction primitive
Table failsA retried top-up is applied twice
Retry semantics and read-only transactions
Service has duplicatesMoney leaves the card but never reaches the wallet
Nothing below you can do many writes at once
Service is missing dataSome customers have a seat but no e-ticket
Transactional outbox · Sagas
Service is missing dataCustomers are charged twice after a service crash
Idempotent consumers: the half the outbox exports
Service has duplicates
Consistency & Isolation Models
After a payment, the user's balance still shows the old amount
Two hierarchies that must not be merged
Service is staleA fix that didn't workOne user sees a payment the sender's history does not list
A consistency model is a promise about histories
Service is staleA transfer received at the same moment vanishes from a balance
The anomalies are three shapes, not five concepts · Read Committed and the lost update
Database is missing dataTwo concurrent withdrawals break a rule both of them checked
Write skew, and the exact boundary of snapshot isolation
Database is wrongEverything said it was fineLocal replica reads are fast but sometimes a second old
Linearizability and sequential consistency · Safety and liveness
Service is staleA fix that didn't workReads from a distant region pay a full round trip to the leader
There is no true state any more
Service is slowTwo devices add points at once and one addition is lost
Eventual consistency, convergence and CRDTs
Database is missing dataA read returns a newer value, and the next an older one
Tunable consistency
Database is staleAfter saving, the old value flips back and forth on refresh
Session guarantees
Service is staleEdits from two devices at once silently lose one
Eventual consistency, convergence and CRDTs
Database is missing dataA capped promotion is overspent while regions are cut off
Highly available transactions and invariant confluence
Database is wrongEverything said it was fine
Query Processing
Two queries with the same answer differ hundreds of times in cost
From what to how: why a query has many plans
Query is slowA sorted query shows nothing for minutes, then every row at once
From what to how: why a query has many plans
Query is slowJoining to a table with two rows per order doubles revenue
Joins, aggregates and windows: what each forces
Dashboard has duplicatesNOT IN returns zero rows after one NULL appears
Bags, NULLs and the operator lattice
Query is wrongA LEFT JOIN filtered on the right side drops unmatched rows
Bags, NULLs and the operator lattice
Query is missing dataA user function in a filter makes a query run for hours
Bags, NULLs and the operator lattice
Query is slowA latest-row-before-a-time join runs for a day
Range, interval and as-of joins
Query is slowOne join task runs for hours while all others sit idle
Broadcast, shuffle and runtime filters
Cluster has a hot spotCOUNT(DISTINCT) runs out of memory where COUNT(*) is quick
Partial, two-phase and incremental aggregation
Query runs outTurning on partial aggregation makes the query slower
The output does not grow with the input
Query is slowA fix that didn't workA large sort is slow with idle disks and the CPU at full
Normalized keys, radix sort and sort avoidance
Query is slowSeveral window functions make the plan sort the table again and again
A sort produces a property, not an answer
Query is slowEach page of LIMIT/OFFSET results is slower than the last
A sort produces a property, not an answer
Query is slowA fast dedup shortcut quietly drops real rows
Set and dedup operations: aggregation with an empty accumulator
Dashboard is missing dataEverything said it was fineA fix that didn't work
Query Planning & Optimization
A report goes from seconds to minutes after a load, nothing changed
The reframe: a layer limited by what the system can know
Query is slowEXPLAIN shows a step expecting a dozen rows that gets millions
The reframe: a layer limited by what the system can know
Query is slowA filter written as ORs scans every partition; the same as IN prunes
Query normalization
Query is slowA minor database upgrade changes a query's plan
Rule-based optimization
Query is slowA query pays for joins whose columns it never selects
A pipeline of one-way steps
Query is slowThe planner expects one row from a freshly loaded partition
Statistics
Query is slowThe same query sometimes takes minutes and sometimes seconds
Join order search
Query never settlesAdding a layer of views makes a query far slower
Join order search
Query is slowWrapping a column in a function makes the query scan every partition
Predicate simplification, constant folding and sargability
Query is slowA CTE used twice is fast in one version and slow in the next
Common subexpression elimination
Query is slowA fresh materialized view is never used by the queries it fits
View expansion and materialized-view rewriting
Query is slowNOT IN takes minutes where NOT EXISTS takes seconds
Subquery decorrelation
Query is slowA query returning a few rows scans terabytes
Metadata-only aggregates and partition pruning · File, row-group and page skipping · Pushdown: negotiation and the ladder · Dynamic partition pruning and runtime filters · Why min/max skipping works or fails completely · Projection pushdown and Top-N pushdown
Query costs too muchOne small dashboard costs more than the whole nightly load
Changing how much data exists
Bill costs too muchOne parameter value takes minutes while the others take milliseconds
Prepared statements and plan caching
Query is slowA join expected thousands of rows and got tens of millions
Adaptive query execution
Query is slow
Execution Engine Runtime
A running job keeps every machine near a quarter of its CPU
Time is work divided by the capacity you kept busy
Cluster is slowA simple in-memory scan is many times slower than a plain loop
Execution models inside one worker
Query is slowOne late task failure restarts the whole job from zero
Execution models across machines
Pipeline failsA serverless run's bill is mostly storage requests, not compute
Execution models across machines
Bill costs too muchDoubling the machines barely shortens the job
Where parallelism comes from and what bounds it
Cluster is slowA fix that didn't workMost cores sit idle at the end while a few tasks finish
Where parallelism comes from and what bounds it
Cluster has a hot spotOne task in a stage takes many times the median
Skew and elasticity
Cluster has a hot spotTurning on speculative execution makes the cluster slower
Skew and elasticity
Cluster is slowA fix that didn't workAutoscaled machines arrive after the slow stage has finished
Skew and elasticity
Cluster is slowA fix that didn't workA shuffle spends its time in millions of tiny reads
Writing shuffle data: hash, sort, spill and consolidation
Cluster is slowA fix that didn't workA large shuffle dies with "too many open files"
The shuffle: a distributed transpose
Cluster failsAfter a network upgrade, shuffle compression makes jobs slower
Shuffle compression, serialization and determinism
Cluster is slowA fix that didn't workAfter a task retry, a total is off by thousands of rows with no error
Shuffle compression, serialization and determinism
Dashboard is wrongEverything said it was fineShuffle disks fill with files from jobs that no longer exist
Shuffle compression, serialization and determinism
Cluster runs out
Memory & Resource Management
The Linux OOM killer kills a nightly job on a large server
Memory cannot be time-sliced
Pipeline runs outWith swap on, the job survives but runs hours longer
Memory cannot be time-sliced
Pipeline is slowA fix that didn't workA JVM service freezes for tens of seconds every few minutes
GC tuning versus manual management
Service is slowProcess memory grows for days while heap use stays flat
Off-heap memory management
Service runs outA fix that didn't workA job dies out of memory after caching every distinct value
String interning and dictionary reuse
Pipeline runs outOne of two identical servers scans markedly slower
NUMA awareness
Cluster is slowAfter a big scan the buffer pool hit rate drops to zero for minutes
LRU and its two failure modes · LRU-K, 2Q and ARC · Scan resistance by declaration
Cache is slowAdding cores makes an in-memory buffer pool slower
Clock: chosen for its hit path, not its hit rate
Cache is slowA fix that didn't workQueries wait for minutes while most of the memory sits idle
Admission control by memory footprint
Cluster is stuckA tiny query is killed out of memory while a huge join survives
Memory grants and reservations
Query failsA user function uses memory that no counter shows
Memory pressure signalling and operator cooperation
Query is unclearSpilling a small share of a join makes it several times slower
What spilling actually costs
Query is slowWhen every query spills at once, everything slows together
Graceful degradation, the honest version · Where spill goes, and what that assumes
Cluster is slowA worker process is killed by the OOM killer mid-batch
Unspillable state and the kill decision
Cluster failsA join on one heavily repeated key never finishes
Sort spill, and choosing the operator by its tail · Hash spill and the arithmetic of a single pass
Query is stuck
Caching
Readers see the old value long after it was corrected
The first copy with no obligation · Freshness bounding
Cache is staleThe database falls over while a restarted cache refills
The governing arithmetic: two inequalities
Database failsCache hit rate stays flat after the cache is doubled
Reuse distance and the miss-ratio curve · Every access stream is three streams · The popularity exponent and the phase transition at α = 1 · Knees and plateaus in the miss-ratio curve
Cache is slowA fix that didn't workp99 stays high although almost every read hits the cache
Miss penalty and the percentile you promised · The tier-insertion rule
Service is slowA remote cache hit is slower than a local SSD read
A distributed cache is not a latency tier · Filtered streams: the same policy at every tier is wrong
Cache is slowA fix that didn't workResult-cache hit rate falls to near zero once a table streams updates
Result caching: the key is the query plus everything it reads that is not data · Materialized views: a cache whose admission was declared
Cache is slowA trivial metadata call takes hundreds of milliseconds
Metadata caching: the one place where the inequality is not close
Query is slowLookups for keys that don't exist all reach the database
Negative caching and Bloom filters
Database is overloadedThousands of requests hit the database when one popular entry expires
Stampede: Little's Law as a count of duplicate work · Warming and prefetch are two mechanisms, not one
Database is overloadedA writer reloads the page and still sees the old value
Versioned keys are MVCC, and the bill goes somewhere new
Cache is staleCache and database disagree after a failed write, and never heal
Write routing, and why you delete rather than update · Freshness bounding
Cache is staleCache hit rate collapses at the same hour every night
Admission beats eviction, and the asymmetry is structural · The eviction policies compared
Cache is slowOne cache node at full CPU while the others sit idle
Consistent hashing, and the hot key that hashing cannot fix
Cache has a hot spotAnother team's use of a shared cache raises load on your database
Cost-aware eviction: the policy family that can be badly wrong
Database is overloaded
Distributed Systems Coordination
A shorter failure timeout makes leaders flap and requests fail
A dead node looks like a slow one · Two halves that never mix: safety and liveness
Database failsEverything said it was fineA fix that didn't workDoubling the partition count leaves the hot partition just as hot
Partitioning and the partition count · Resharding: the map is harder than the bytes
Database has a hot spotA fix that didn't workA rebalance takes over a day and fills the source disks
Rebalancing: a controller with a loop delay measured in hours
Database runs outUsers far from the leader see old values
Read replicas and follower reads: the cost ladder
Service is staleA paused leader wakes and accepts writes the new leader already took
Locks, leases and fences: why mutual exclusion is unavailable · Leader election and epochs: one mechanism, many names · Clocks: three requirements · Failure detection: liveness is a relation, not an attribute
Database has duplicatesThe cluster briefly has two leaders with no network failure
Membership, reconfiguration and discovery · Consensus
Cluster failsClients keep calling a node removed an hour ago
Service discovery and load balancing
Service failsA fan-out page's p99 is far worse than any backend's
Scatter-gather · Hedging
Service is slowWork for abandoned requests turns a brownout into an outage
RPC: idempotency and deadlines
Service fails
Fault Tolerance & Delivery Semantics
Customers charged twice for one purchase
Why failure recovery is a different kind of problem · Epochs and sequence numbers: one number instead of a dedup set · Idempotency: the three sources
Service has duplicatesSome customers are never billed although the work was done
Why failure recovery is a different kind of problem
Service is missing dataHundreds of clients retry together and flatten the database
Retry with exponential backoff and jitter
Database is overloadedAn hour after a restart, the backlog has barely shrunk
Replay from log and the catch-up condition
Stream is slowTwo machines both act as primary for a while
Failover: automatic, manual, fenced
Database failsA fix that didn't workA job of thousands of tasks never finishes without some failing
Task re-execution and blacklisting
Pipeline failsReplaying a day gives totals close to, but not equal to, the original
Replay from log and the catch-up condition
Dashboard is wrongOne malformed message blocks a log partition for hours
Poison pills and skip-and-log
Stream is stuckCustomers receive the same notification twice
Consumer offsets and commit ordering · At-most-once, at-least-once and the exactly-once category error
Service has duplicatesTwo of three replicas sit in the rack that loses power
Replication factor and quorums as a survivability budget
Database fails
Emergent Failure Dynamics
A few seconds' stall is followed by many times the normal load
Retry amplification
Service is overloadedLoad spikes at exactly the same clock time
Thundering herd
Service is overloadedThe database collapses after a cache restart
Cache stampede (dynamics only)
Database failsRouting to the least-loaded server makes the overload worse
Least-loaded routing
Service is overloadedA fix that didn't workA queue in front of a service makes every request late
Queue amplification and unbounded buffering
Service is slowA few seconds' stall becomes a long outage
Congestion collapse · The loop as an object
Service failsThe site fails again within a minute of a full restart
Why restarting sometimes works
Service failsA fix that didn't workThe outage continues after the cause is rolled back
Trigger versus sustaining effect
Service failsA fix that didn't workShedding a fifth of the traffic does not bring the site back
The formal condition
Service failsA fix that didn't workA service stops answering everything when one optional call slows
Resource-exhaustion cascade · Dependency cascade · Timeout cascade
Service failsOne configuration change takes down every region at once
Shared fate · Blast radius, failure domains and cells
Service failsThe autoscaler cannot add capacity during the outage it should fix
Static stability and the control-plane asymmetry
Cluster is stuckA fix that didn't workEvery instance stops in the same second, across all zones
Shared fate
Service failsEvery request is slow when one node of many is slow
Why fail-slow is worse than fail-stop
Cluster is slowEvery node fails its health check at once and the fleet is evicted
Differential observability
Cluster failsA job reports success while one segment's rows are missing for days
Gray failure in the data plane
Pipeline is missing dataEverything said it was fineOne node runs far slower than its identical peers
Fail-slow hardware
Cluster is slowA node passes its health check while running far slower
Fail-stop as a purchased assumption
Cluster is slowEverything said it was fineHeap fills and GC pauses grow while traffic stays flat
GC death spiral
Service is slowCompaction falls behind and read latency doubles at flat traffic
LSM compaction debt
Database is slowThe autoscaler adds and removes servers every few minutes
Autoscaling oscillation
Cluster never settlesThe system falls over again as queued work replays after an outage
Backlog burst and replay storms
Service failsA recovered system collapses again while its caches are empty
Cold-cache collapse
Service fails
Scheduling & Workload Management
Every job ends up high priority and waits are unchanged
From how to which
Cluster is slowA fix that didn't workA two-second query waits most of an hour behind a big report
FIFO, priority and multi-level feedback
Query is slowA guaranteed share of the cluster takes an hour to arrive
Fair and capacity scheduling · Preemption
Cluster is stuckThe cluster is partly idle, yet a job stays pending for hours
Gang scheduling · Backfill scheduling
Cluster is stuckThe same job runs twice as slow on some days
Locality: node, rack, zone, any
Cluster is slowAll three replicas go down together when one rack loses power
Rack and zone awareness · Anti-affinity · Bin packing versus spread placement
Service failsA cheaper spot cluster loses a pipeline mid-run
Cost-aware placement and spot
Cluster failsA fix that didn't workCPUs sit half used while no new task can start
Heterogeneous resource matching
Cluster is stuckThe scheduler runs at full CPU while the machines sit idle
The scheduler is shared state
Cluster is stuckThe queue holds thousands of queries whose users already left
Admission control and queuing
Cluster is overloadedAn accidental query runs for an hour, then fails near the end
Runaway killers and result governors
Query failsOne team's heavy jobs slow everyone, though all are within quota
Multi-tenant isolation and quotas
Cluster is slowEverything said it was fineAutoscaled machines arrive after the burst is over
Autoscaling
Cluster is slowA fix that didn't workCPU is fully busy and almost nothing useful finishes
Concurrency limits and slots
Cluster is slowDashboards that answer in seconds take over a minute at peak
Prioritization and short-query lanes
Query is slow
Cost Mechanics
A colder storage tier costs far more than size times its low rate
Storage: an integral over time · Per-request pricing: the envelope and the letter
Bill costs too muchA fix that didn't workAn engine cheaper per hour raises total compute spend
Compute: capacity-time, not work
Bill costs too muchA fix that didn't workA job of a few seconds is billed as a full minute
Minimums, labour and the price trajectory
Bill costs too muchThe bill is several times the forecast though every rate matches
The four things a vendor can meter · What pricing forbids, and the three escapes
Bill costs too muchRequest fees run several times the storage fees on a bucket
Target file size · LIST cost and manifests
Bill costs too muchOne query costs thousands of times a near-identical one
The computable value of pruning · Materialization break-even
Query costs too muchA shuffle spread across zones adds a surprising network charge
Cross-AZ charges and placement · Egress and data gravity
Bill costs too muchA scale-to-zero warehouse costs more than a small always-on one
Scale-to-zero: a one-line break-even
Bill costs too muchA fix that didn't workA job on spot machines is interrupted every night and never finishes
Spot capacity: the vendor's idle stock, resold
Cluster is stuckA fix that didn't workPart of a long-term commitment sits unused, with no alarm
Commitment discounts: you sell flexibility
Bill costs too muchA fix that didn't workA schedule change multiplies a query's scan cost and nobody notices
Governors and runaway kills · Cost-based routing and result caching
Bill costs too muchOne team's cost falls while the total rises
Tagging and attribution
Bill costs too muchA fix that didn't workA warehouse bills through idle minutes before it suspends
Idle warehouses and SELECT *
Bill costs too muchA streaming job writes a new file every few seconds per partition
Chatty requests, needless recompute and cross-region calls
Table costs too much
Time & Temporal Semantics
Re-running the same code on the same events gives a different count
Three clocks: which time is on the record?
Dashboard is wrongSome events are stamped 1970, others years in the future
Three clocks: which time is on the record?
Dashboard is wrongAn hourly window never closes and no result appears
Watermarks: declaring completeness
Stream is stuckAn unrelated operator's memory grows without limit
Watermarks: declaring completeness
Stream runs outAfter allowing late updates, the hour's total doubles
Triggers and accumulation mode
Dashboard has duplicatesA fix that didn't workEvents from offline devices arrive late and are dropped silently
Out-of-order handling and buffering
Stream is missing dataEverything said it was fineA live count rises, falls, then rises again
Late data: drop, divert or re-fire
Dashboard never settlesA folder already used for payments changes days later
Retractions, upserts and changelog semantics
Dashboard is wrongA one-day query reads many days of folders
Which clock keys the partition
Query is slowTraining data joins each event to today's value, not that day's
Temporal joins and future leakage
Dashboard is wrongNobody can say what a report said on an earlier date
Bitemporal modeling and time travel
Dashboard is unclearA short timeout waits almost an hour after the clock is corrected
Monotonic clocks, wall clocks and replay determinism
Service is stuckA replay after an outage counts deliveries twice
TTL and expiry
Dashboard has duplicatesDuplicate local-time keys appear on the night the clocks go back
Time zones and calendars
Table has duplicatesA backfill of last week gives different totals from the original run
Time zones and calendars
Dashboard is wrongDate arithmetic gives different answers depending on the order
Time zones and calendars
Query is wrong
Streaming
A job crashes out of memory after weeks at flat traffic
The governing question and its five answers
Stream runs outThe whole job freezes for seconds every few minutes
State backends: which problems you import
Stream is slowA fix that didn't workMore parallelism changes nothing; one instance runs at full CPU
Keyed state, operator state and the indivisible key
Stream has a hot spotA fix that didn't workJob state keeps entries for users idle for years
State TTL and expiry
Stream runs outRestarting a stopped source makes the job emit a burst of results
Triggers, evictors and the timer service
Stream is overloadedOne idle source partition stops every window from closing
Watermark propagation
Stream is stuckA pattern-matching rule's state grows faster than any aggregation
Complex event processing
Stream runs outA long sliding window pins the CPU and holds huge state
Windows are retention rules in time
Stream is slowA per-record service call caps throughput, whatever the machine count
Stream-table joins and enrichment
Stream is slowA fix that didn't workA stream join matches a few percent fewer rows, with no error
Stream-stream joins
Dashboard is missing dataEverything said it was fineRe-running last month with the same code gives different numbers
Stream-stream joins
Dashboard is wrongA running maximum never drops after a cancellation
Changelogs and table-stream duality
Dashboard is wrongA simple MAX keeps a huge state
Changelogs and table-stream duality
Stream runs outAfter a fast restore the job takes over an hour to catch up
Incremental checkpoints and recovery time
Stream is staleWith exactly-once on, results arrive late and some go missing
Exactly-once: checkpoint plus transactional sink
Stream is missing dataEverything said it was fineA fix that didn't workCheckpoints grow from seconds to minutes, worst in busy hours
Checkpointing: a consistent cut
Stream is slowAn autoscaling script scales up and down, never settling
Dynamic scaling of stateful operators
Stream never settlesA fix that didn't workA new version refuses to start: it cannot read the old state
Savepoints and state schema evolution
Stream failsRescaling a stateful job takes minutes of downtime, then lag
KeyBy and key groups
Stream is slowBackpressure shows red on several operators; the culprit is unclear
Backpressure propagation
Stream is unclearA streamed total falls short of the warehouse while all looks healthy
What streaming forbids, and what the escapes cost
Dashboard is missing dataEverything said it was fine
Change Data Capture & Incremental Processing
An incremental pipeline runs slower than the full rebuild it replaced
The arithmetic that governs everything
Pipeline is slowA fix that didn't workSource database disk fills after a change reader stops
Log-based CDC
Database runs outIntermediate status changes never reach the copy
Trigger-based CDC
Dashboard is missing dataRows deleted at the source still count in the copy
Polling and the completeness gap
Dashboard is wrongValues fixed by a bulk script never reach the copy
Polling and the completeness gap
Dashboard is staleA downstream job fails with "snapshot not found" after a long pause
Snapshot-diff incrementality
Pipeline failsAn incremental job skips data that arrives late for a finished day
Watermark-based incremental batch
Pipeline is missing dataA transfer shows money gone from one account, not yet in the other
Ordering and transactional grouping
Dashboard is wrongA consumer crashes after a source column changes type
Schema change propagation
Stream failsSome rows show old values after the initial copy
Bootstrap: snapshot and streaming handoff
Dashboard is staleA merge that changes few rows rewrites almost every file
The application granularity ladder
Table is slowA maintained view keeps showing a deleted row's value
Incremental view maintenance
Dashboard is wrongA history table has overlapping, missing and impossible date ranges
Feeding history tables
Dashboard is wrongA merge fails on duplicate keys, and the workaround leaves duplicates
Keys, and why upsert presupposes them
Pipeline has duplicatesA row whose key changed at the source shows up twice in the copy
Keys, and why upsert presupposes them
Dashboard has duplicatesA few writes reach the database but never publish their message
Replay and dual-write avoidance
Stream is missing dataReplaying the log leaves older days still wrong
Replay and dual-write avoidance
Dashboard is wrongA fix that didn't workA day of rows goes missing on switching to a backfilled table
Backfill and cutover
Dashboard is missing dataA fix that didn't workTotals fall slightly short while every pipeline is green
What change data capture forbids, and what escaping costs
Dashboard is missing dataEverything said it was fine
Table & Storage Maintenance
A dashboard slows over months with the same query and similar data
The first work nobody requested
Query is slowEverything said it was fineA table holds millions of tiny files for its size
Why small files appear · Compaction as bin-packing
Table is slowCompacting small files makes dashboards slower
When packing ruins clustering, and defragmentation
Query is slowA fix that didn't workStorage is several times the size of the live table
Snapshot expiration: history costs space
Bill costs too muchA long read-only report fails with "file not found" after a cleanup
Vacuum: reachable is not in use
Query failsA fix that didn't workMoving old data to a colder tier raises the bill
Lifecycle tiering
Bill costs too muchA fix that didn't workCompaction of the busiest partition fails with a conflict every night
Concurrency: maintenance competes with writers
Table fails
Data Modeling
Two dashboards show different revenue for the same quarter
A layer with no oracle
Dashboard is wrongEverything said it was fineA sum counts some days twice because of subtotal rows
Grain: one row equals one what?
Dashboard has duplicatesTeams put the same country in different regions
Conformed dimensions, the bus matrix, role-playing dimensions and reference data
Dashboard is wrongLast year's report shows different numbers when rerun today
As-was or as-is: the slowly changing dimension types
Dashboard is wrongA balance summed over the year is many times anything ever owed
Fact table types
Dashboard is wrongAn average margin reads far higher than finance's number
Additivity comes first
Dashboard is wrongA per-person leaderboard adds up to more than total revenue
Many-to-many: bridge tables, fan traps and chasm traps
Dashboard has duplicatesA total after a join is more than double the invoices
Many-to-many: bridge tables, fan traps and chasm traps
Dashboard has duplicatesThree source systems give the same customer three addresses
Data Vault
Dashboard is wrongOne person appears under three emails
Entity resolution
Dashboard has duplicatesThree teams report three different counts of active customers
The semantic (metrics) layer
Dashboard is wrongFinding records by an item inside an array scans everything
Nested and repeated modeling
Query is slowMoving one branch of a hierarchy rewrites thousands of rows
Hierarchies
Table is slowOlder days can't be fixed because the source files are gone
Medallion layering
Dashboard is wrong
Schema Management & Evolution
A field rename zeroes a dashboard while every pipeline is green
Structure becomes a variable
Dashboard is missing dataEverything said it was fineAfter a column is added, every price downstream looks wrong
Identity by position
Dashboard is wrongEverything said it was fineAn inferred schema flips a column's type from one week to the next
Schema inference and drift
Pipeline failsTwo dashboards parse the same raw events into different counts
Schema-on-write versus schema-on-read
Dashboard is wrongA unit change keeps the type, so every check passes on wrong values
Type promotion versus coercion · Contracts, required fields and semantic types · CI-time schema diff
Dashboard is wrongEverything said it was fineA new status value silently falls into "Other"
Enum and symbol-set evolution
Dashboard is wrongAdding an optional field crashes an old client
Compatibility modes from deployment order
Service failsNearly half the records go to a dead-letter queue nobody watches
Dead-letter versus permissive parsing
Pipeline is missing dataEverything said it was fineA schema registry outage stops every producer
Registry and subject strategies
Stream failsA new column appears in many tables overnight and breaks reports
Propagation through a derived DAG
Table failsNobody can say what a change to a table would break
Versioned schemas and migration
Table is unclear
Data Quality & Validation
Adding checks after every step makes the nightly run much longer
What makes a check expensive
Pipeline is slowA fix that didn't workEvery price jumps a hundredfold and no check notices
Propositions about the world
Dashboard is wrongEverything said it was fineOrders point to customer IDs that never existed, and type checks pass
Schema validation and the per-value tier
Dashboard is wrongEverything said it was fineA declared primary key does not stop duplicate IDs
Declarative constraints: why the warehouse stopped enforcing them
Table has duplicatesEverything said it was fineA volume check fires every Sunday and every month-end
Statistical checks: from predicates to estimators
Pipeline raises false alarmsThousands of generated checks raise daily alerts that get muted
Multiple comparisons and alert fatigue
Pipeline raises false alarmsA fix that didn't workA check fires every night and clears itself by noon
Cross-field and cross-table rules
Pipeline raises false alarmsAn empty input file passes every check
Statistical checks: from predicates to estimators
Pipeline is missing dataEverything said it was fineA batch publishes with rows set aside and nobody is paged
Quarantine: how a loud failure becomes a quiet bias
Dashboard is missing dataEverything said it was fineA fix that didn't workStopping the pipeline on a failed check leaves hours of backlog
Stop-the-line, and why the factory analogy brings in the wrong cost model
Pipeline is slowA fix that didn't workDeduplication merges thousands of different people into one
Deduplication: three different problems in one word
Dashboard is wrongA fix that didn't workLoosening a match threshold a little makes merges far worse
Deduplication: three different problems in one word
Dashboard is wrongA fix that didn't workTwo systems are compared every night and nobody can say which is wrong
Cross-system audits
Dashboard is unclearA full row-by-row comparison takes hours and only says "different"
The reconciliation ladder: detecting is cheap, locating is what costs
Pipeline is slow
Metadata, Catalog & Lineage
The catalog lists an owner who left over a year ago
Load-bearing versus advisory
Table is staleA table's catalog description is years out of date
Technical catalog versus business glossary
Table is staleA query spends many seconds planning before reading any data
Manifest and snapshot trees · Metastore mechanisms
Query is slowA frequent streaming writer starts failing with commit errors
The commit point
Table failsAfter a rename, a table's lineage and usage history are empty
Asset identity: what is the row keyed on?
Table is missing dataA cleanup bot proposes deleting a busy table after its rename
Asset identity: what is the row keyed on?
Table is wrongRepairing two days of bad output means reprocessing a month
Provenance annotations
Pipeline is slowA fix that didn't workA schema service outage stops every consumer
The schema registry as metadata
Stream failsThe planner picks a terrible plan on the fastest-growing table
Statistics stores
Query is slowLineage says no path exists, but the data flows anyway
Fidelity and the absence asymmetry
Access is exposedEverything said it was fineCatalog search ranks a deprecated table first
Search, discovery and ranking
Table is wrongA table unused for months is deleted, then needed once a year
Usage and query-log mining
Table is missing dataA fix that didn't work
Orchestration & Workflow
A step exits with code 0 after processing an empty file
Orchestration moves nothing
Pipeline is missing dataEverything said it was fineFinding where a pipeline stands means reading logs in four systems
Orchestration moves nothing
Pipeline is unclearA run labelled one day starts the day after
Cron, calendars and data intervals
Pipeline is wrongThe dashboard shows yesterday's numbers while the run is green
DAGs, dependency resolution and trigger rules
Dashboard is staleEverything said it was fineAll workers are busy and nothing is running
Sensors: polling versus push
Pipeline is stuckTwo copies of a retried task write one partition; it matches neither
Retries, the unknown state and fencing · Idempotent task design and completion verification
Dashboard is wrongUnpausing a workflow launches hundreds of backfill runs at once
Backfills, catchup and versioning
Pipeline is overloadedA fix that didn't workA run is green and produced nothing
Branching and short-circuit
Pipeline is missing dataEverything said it was fineA fan-out of thousands of tiny tasks slows the orchestrator to a crawl
Dynamic task generation
Pipeline is slowNo run starts for days and everything stays green
SLAs and absence detection
Pipeline is missing dataEverything said it was finePipelines starting together take every warehouse slot
Pools, max active runs and depends_on_past
Cluster is stuckA sensor watching a renamed folder never fires again
Cross-DAG and cross-system dependencies
Pipeline is stuckA deployment restart loses a request waiting for approval
Human-in-the-loop steps
Pipeline is missing dataRe-running a failed run behaves differently from the original
Parameterized runs and run configs
Pipeline is wrongThe scheduler takes minutes to notice that tasks have finished
Scheduler self-durability and scale
Pipeline is slowA top-priority pipeline still finishes late, its queries queued
Priority and starvation across pipelines
Pipeline is slowA fix that didn't workA long backfill never completes behind production runs
Priority and starvation across pipelines
Pipeline is stuck
Observability & Operations
Monitoring goes down with the cluster it watches, and no alert fires
The telemetry pipeline is a data system
Service failsEverything said it was fineRevenue is far too low for days while every dashboard is green
Observation is a second machine
Dashboard is wrongEverything said it was fineOne new metric label doubles the bill and exhausts the metrics server
Cardinality and the four signals
Service runs outNobody can tell which record broke the job
Structured logging
Pipeline is unclearA p99 panel reads fine while customers wait seconds
Metrics: counters, gauges and histograms
Service is slowEverything said it was fineA job succeeds over empty input and the table goes stale
Freshness
Pipeline is staleEverything said it was fineConsumer lag looks small while downstream is hours behind
Consumer lag and skew
Stream is staleEverything said it was fineA job's progress sits at 95% for most of an hour
Correct per arrival, and progress
Pipeline is stuckMost pages in an on-call week are noise
Anomaly detection and alert fatigue
Service raises false alarmsAfter a blaming review, people stop sharing detail in incidents
The incident lifecycle
Service is unclearA real page is missed after many false ones
Alerting is a base-rate problem, not a detection problem
Service raises false alarmsAn expected file never arrives and no signal moves
Canaries and synthetic transactions
Pipeline is missing dataEverything said it was fine
Security & Privacy
A pen test gets in although every compliance check is green
The first adaptive adversary
Access is exposedEverything said it was fineA leaked token is used from outside and stays valid for weeks
Authentication
Access is exposedThe access system grows to thousands of roles
Authorization models
Access is unclearGrants from finished projects are still active
Views, grants and privilege inheritance
Access is exposedA user limited to a view reads every row through the storage files
Short-lived credentials and vending
Access is exposedA fix that didn't workThe audit log shows one shared account, never who asked
Delegation chains and egress
Access is unclearA row-filtered query scans the whole table, and the bill shows it
Row- and column-level security
Query costs too muchQueries on an encrypted column run many times slower
Encryption at rest and envelope encryption
Query is slowA fix that didn't workA masked column can still be probed with filters
Masking and tokenization
Access is exposedA fix that didn't workHashed identifiers are reversed in minutes
Hashing and salting identifiers
Access is exposedA fix that didn't workA news story, postcode and birth year identify a person in an extract
K-anonymity and its successors
Access is exposedEverything said it was fineA fix that didn't workCombining two anonymized extracts identifies more people
Differencing and differential privacy
Access is exposedAn erasure request finds the person's data in a dozen places
Data minimization and the right to be forgotten
Access is exposedA departed person's account still works and is used
The identity gaps
Access is exposedA breach goes unnoticed for weeks and no alert fires
Detection and accountability
Access is exposedEverything said it was fineMillions of rows read through valid access and copied out
Egress, exfiltration and multi-tenancy
Access is exposedEverything said it was fineOne user receives another user's query results
Egress, exfiltration and multi-tenancy
Access is exposed
Specialized Processing Models
A word search inside titles reads every row
The one assumption behind every ordinary index
Query is slowA bounding-box filter misses points just across its edge
The one assumption behind every ordinary index
Query is missing dataA brand search returns unrelated items and no setting fixes it
Analyzers and tokenizers
Query is wrongAdding one common word makes a search hundreds of times slower
Scoring and top-k skipping
Query is slowShowing category counts makes a results page many times slower
Phrase queries and faceting
Query is slowA new item is invisible in search for up to a minute
Segments and near-real-time refresh
Service is staleShortening the search refresh delay slows the whole cluster
Segments and near-real-time refresh
Service is slowA fix that didn't workA better embedding model makes similar-item results worse
Embeddings, and why exact indexes fail
Query is wrongA fix that didn't workA filtered vector search returns far fewer results than exist
Hybrid search and filtered ANN
Query is missing dataA fix that didn't workExact similar-item search takes half a second per query
Nothing can be eliminated by inspection
Query is slowA three-hop query never finishes for users near huge accounts
Traversal and degree skew
Query is stuckAdding replicas does not speed up one slow graph query
The access pattern is discovered, not declared; CSR
Database is slowA fix that didn't workA point just across a tile boundary is never found nearby
Filter-and-refine and spatial partitioning
Query is missing dataA zone that straddles two partitions is counted twice
Spatial joins and discrete global grids
Dashboard has duplicatesTwo geometry libraries disagree on whether a point is inside
Geometries and coordinate reference systems
Query is wrongMetrics storage grows fast though most points repeat the last value
Compression as the write path
Bill costs too muchLast quarter's hourly p99 can't be computed from stored averages
Downsampling, rollups and retention tiers
Dashboard is unclearA new label makes the metrics store run out of memory
Series identity and the cardinality wall
Service runs out
Interfaces & Formats
A column rename breaks an outside consumer for weeks unnoticed
What changes at a boundary
Pipeline failsEverything said it was fineSplitting a file at byte offsets cuts rows in half
Text formats: CSV, JSON and XML
Pipeline is wrongA fix that didn't workTotals are off by a few cents though every file loaded cleanly
The interop hazards
Dashboard is wrongEverything said it was fineA load job uses one worker of many for hours
Where does the schema live?
Cluster has a hot spotExporting a huge result through a SQL driver runs for hours
JDBC, ODBC, REST and GraphQL
Query is slowA webhook charges a customer twice after a slow reply
Catalog APIs and webhooks
Service has duplicatesA long transfer fails near the hour with "unauthorized"
A conversation has state; SQL as the interface
Pipeline failsA push into the CRM takes days and leaves records half updated
Federation, mounting, zero-copy sharing and reverse ETL
Pipeline is slowPaging through an API sync slowly loses records, with no error
Pagination, cursors and incremental sync tokens
Pipeline is missing dataEverything said it was fineA five-year backfill through an hourly API would take weeks
File drop, backfill, and what integration forbids
Pipeline is slowA sync by last-updated timestamp loses rows at page boundaries
Connector frameworks, ELT and ETL
Pipeline is missing data
Measurement & Experimental Method
A benchmark reports a rate too good to be true
Predictions, hypotheses and variables
Service is wrongA benchmark win shrinks and then reverses on fresh machines
Pinning, isolation and cold versus warm
Service never settlesA recommended setting is worse than the default on your machine
Workload realism and scale ladders
Service is slowA fix that didn't workA one-second freeze shows up as a single slow sample
Coordinated omission and steady state
Service is slowEverything said it was fineA service that passed its load test collapses under real traffic
The harness is not neutral: open versus closed loop
Service failsEverything said it was fineAveraged per-shard p99s look fine while users time out
Percentiles do not compose; HDR histograms
Service is slowEverything said it was fineThe dashboard shows full throughput while almost no user succeeds
Throughput at an SLO, and goodput
Service failsEverything said it was fineA CPU profile of a slow request is flat
Measurement overhead and wall versus CPU time
Service is unclearMean latency looks low while users time out
Recording is irreversible; distributions, not averages
Service is slowEverything said it was fineThe best of many tuned configurations gains far less in production
Inference, run counts and the two variances
Service is wrongA fix that didn't workA slower path shows a flat profile with cores mostly idle
Sampling, instrumentation and hardware counters · The USE method and per-operator profiles
Service is unclearIn another region the measured advantage disappears
Publishing configs and one-command reruns
Service never settlesEvery test ran at a fraction of the production data size
Threats to validity and scale honesty
Service is unclear