The mechanisms underneath every data system.
From the physics of disks and networks to the method of measuring them. Each mechanism answers one question, and its parts come in the same order.
BandThe Machine: jump to another band
The physical and arithmetic limits every system inherits: disks, caches, networks and queueing laws.
How data travels between places: transport patterns, flow control, routing and the wire.
How data is laid out, made durable, structured, encoded and indexed once it lands.
Storage: Physical Organization
What should sit next to what, given that I must decide at write time and find out at read time?
Layout models · Mutation strategies · Physical clustering · Access & media
Persistence & Durability
When a system says "yes, your data is safe", what exactly has it promised, and what did it have to do to keep that promise?
Write-ahead log · Group commit & fsync · Checkpoints & snapshots · Replication & quorums
Core Data Structures
How do we choose a data structure when the real cost is set by the hardware, not by Big-O?
Lookup structures · Ordering structures · Probabilistic sketches · Bit-level structures · Concurrency-friendly structures
Encoding, Serialization & Compression
What do we turn our values into when we turn them into bytes, and what does that choice cost?
Serialization · Lightweight columnar compression · General-purpose compression
Indexing
When does a separate "where is it?" structure pay for itself, and what does it cost to keep it true?
Primary & secondary indexes · Covering indexes · Inverted & vector indexes · Index maintenance
What readers and writers are promised when their work overlaps.
Transactions & Concurrency Control
What happens when many users change the same data at the same time, and how do we keep the result correct?
Concurrency control schemes · Transaction mechanics · Distributed transactions
Consistency & Isolation Models
When many users, many objects and many copies exist at once, which histories does the system promise never to show, and who writes the code for the histories it does show?
Isolation levels · Distributed consistency models
How a question becomes a plan, how the plan runs, and what happens when it does not fit in memory.
Query Processing
Why can one question be answered by plans whose costs differ by a factor of ten thousand, and what does each operator force the engine to do?
Relational operators · Join algorithms · Aggregation · Sorting · Set & dedup
Query Planning & Optimization
How does a system choose a plan for a question when the facts that decide the best plan are unknown until the plan has run?
Planning · Data-reduction optimizations · Execution optimizations
Execution Engine Runtime
Once a query plan is chosen, why does the machine still sit idle, and what does each mechanism of a query runtime pay to buy that idle time back?
Execution models · Parallelism · Shuffle
Memory & Resource Management
Who is allowed to hold how much memory, for how long, and what happens to the work when the answer is less than it asked for?
Buffer pools · Memory grants · Spill-to-disk · Off-heap memory
Caching
What happens when we deliberately keep a second copy of data that no protocol forces to be correct?
Result & metadata caching · Materialized views · Invalidation · Eviction policies
How many machines act as one, survive failure, and break themselves under load.
Distributed Systems Coordination
Who is allowed to decide, and how does everyone else learn the decision without asking?
Distribution of data · Coordination · Communication
Fault Tolerance & Delivery Semantics
What is the cheapest thing that can be safely done twice?
Recovery mechanisms · Correctness under failure · Availability
Emergent Failure Dynamics
What happens when nothing is broken, every component is healthy, and the system's own response to stress makes the stress worse?
Feedback loops · Metastable failure · Cascading failure · Gray failure · Death spirals · Overload defences
Who gets capacity, when, and what every choice costs in money.
Scheduling & Workload Management
Which work runs now, which work waits, and who pays for the waiting?
Job & task scheduling · Placement · Workload management
Cost Mechanics
What is metered, and what is the smallest increment you are charged for?
Pricing primitives · Pricing → design · Elasticity · Attribution · Cost anti-patterns
Data that keeps arriving, keeps changing, and must be kept healthy as it ages.
Time & Temporal Semantics
When am I allowed to act on data that may still be incomplete, and what do I owe the reader if I was wrong?
Event vs. processing time · Watermarks · Late data · Triggers · Bitemporal data
Streaming
What does it cost to remember, when the input never ends?
Stateful operators · State backends · Windowing · Stream joins · Checkpoints
Change Data Capture & Incremental Processing
What does it cost to know what changed, and is that cheaper than not needing to know?
Log-based CDC · Snapshot handoff · Upsert & merge · Incremental view maintenance · Backfills
Table & Storage Maintenance
Who pays for the gap between the order data arrives in and the order it is read in: the writer, every reader, or a background job, and what is the break-even?
Compaction · Small files · Clustering · Vacuum · Snapshot expiration
How data carries a definition its consumers can rely on: modeling, schema evolution, quality and metadata.
Data Modeling
What does one row mean, who is allowed to change that answer, and what breaks downstream when they do?
Star schema · Slowly changing dimensions · Data Vault · Semantic layer · Entity resolution
Schema Management & Evolution
What happens when the shape of your data changes on its own schedule, and nobody can change the readers, the writers and the old data at the same moment?
Schema-on-write vs. read · Compatibility modes · Column mapping · Data contracts
Data Quality & Validation
How do you check that data is right, when "right" is a claim about the world and not about the bytes?
Validation · Consistency & reconciliation
Metadata, Catalog & Lineage
What can a system prove about its data without reading the data, and how does it know that proof is still true?
Catalogs · Metadata trees · Lineage graphs · Data versioning
How pipelines are coordinated, observed and protected in production.
Orchestration & Workflow
How do you make unreliable, unobservable, irreversible side effects behave like a pure function of a time interval?
DAGs & dependencies · Scheduling & triggers · Backfills · Idempotent tasks · Durable execution
Observability & Operations
What is the smallest description of a running system from which we can find a fault, fast enough to matter, at a cost we can still pay while the system is failing?
Metrics, logs & traces · Query profiling · Freshness monitoring · Alerting
Security & Privacy
How do you protect data from someone who watches your design and then chooses how to attack it?
Identity & access · Protection · Privacy mechanisms
Workloads that aren't rows and ranges (text, vectors, graphs, geo, time series) and the contracts at system edges.
Specialized Processing Models
What do you build when a query predicate cannot be reduced to a range over an ordered key?
Text search · Vector search · Graph · Geospatial · Time series
Interfaces & Formats
What can two parties agree on when they cannot talk to each other?
File formats · Access protocols · Integration patterns
How to tell whether a measurement means what you think.