Someone curious enough to understand how data systems work underneath.
I’ve spent five years building data pipelines. This is where I work through the mechanisms underneath them from first principles, with little labs you can run to see each one for yourself.
Aayush Sharma, a data engineer with five years of building pipelines in fintech. I write the material and run every lab before it is published.
For data engineers who have watched a system do something they couldn’t explain: a job that got slow with no code change, a count that was off by a few rows. If you want to learn a tool step by step, its documentation is the better place to start. Come back when it surprises you.
A few years ago, knowing a tool’s API was the skill. Now a model writes that code in seconds, and it is usually close. What it doesn’t give you is the sense that “close” is wrong here: the join that spills at ten times the data, the retry that writes a payment twice. That comes from knowing what happens underneath.
I write the material; AI helps turn it into pages, structure, links and visuals. Every lab is run before it is published, and the numbers on a lab page are rendered from that run, never typed. The cases are constructed illustrations: the companies and people are made up, the mechanisms are not.
33 mechanisms, from storage media and encoding to transactions, streaming, cost and measurement. Each one answers a single question and is read the same way: how it works, from first principles; a constructed case, drawn from everyday data work, that opens and closes every article; playbooks for the decisions it forces; labs you run yourself; and interview questions that test the reasoning.
Underneath, the pieces are connected: 46 trade-offs and principles traced across mechanisms, 193 relationships with their formulas charted, and 2,350 glossary terms explained where you meet them. Routes give you an order to read in.
Systems: real tools taken apart as bundles of mechanism choices.