Security & Privacy
How do you protect data from someone who watches your design and then chooses how to attack it?
Who this mechanism is for: a developer who writes code, calls APIs and knows basic SQL, and who now has to keep data safe. You do not need to know cryptography, security standards or privacy law. Every technical term is explained the first time it appears, and again in the glossary at the end. This mechanism works with a data platform (files in cloud storage, tables, SQL engines, pipelines), because that is where security and privacy become hard.
The story we will follow
Hollis Health is a regional hospital network: 9 hospitals, 40 clinics and about 2.1 million patients. Its data platform is called the Lake. The Lake is a lakehouse: data files (in a format called Parquet) sit in cloud object storage, a catalog keeps a list of which files form which table, a SQL engine answers queries, nightly pipelines move data around, and analysts use dashboards and notebooks. Tables include patients, lab_results, visits, billing and device_readings. Some columns are very sensitive: national_id, date_of_birth, postcode, diagnosis_code, phone.
Wei Zhang is the data engineer who owns the Lake. Ingrid Halvorsen is the head of security; she never accepts the answer "it is encrypted" and always asks "against whom?" Mateus Silva is an analyst in the Northern region finance team: honest, always in a hurry, and quick to find a workaround when a control is slow. Later we meet Dr. Adaeze Eze, the privacy officer, who also practises as a doctor.
Hollis has an external audit coming. Each article of this mechanism opens with a problem that looks like a bug or a misconfiguration. Each one is actually a structural property of security work. At the end of the article, we go back and solve it with the article's mechanisms.
This is the main idea of this whole mechanism:
How this mechanism is organized
- 1
Why security is different
Why can security never be judged like latency or cost, and what are the three places we can intervene?
CaseEvery box green, and a pen-tester walked in
- 2
Identity and access
How do we prove who a caller is, and how do we store the rules about who may do what?
CaseA leaked token with 54 days left, and 2,400 roles
- 3
Enforcement points
Where must the rules be enforced so that they are real, and how do credentials, views, grants and delegation behave?
CaseEvery region visible through the files
- 4
Protection
What can we do when the adversary already holds the bytes (encryption, masking, tokenization, hashing)?
CaseAn encrypted column, a 40-minute query
- 5
Privacy
What do we do when every access was authorized and the data still leaks?
CaseA 5-anonymous extract with a named politician in it
- 6
Where it breaks
Which assumptions underneath the mechanisms does nobody own, and how is a failure found?
CaseExposed for 11 weeks, and every control green
- 7
What security forbids
What can no combination of controls give you, and what position is left to choose?
CaseA security statement for the board
After that: a summary, hands-on labs, interview questions and a glossary.
One request: for every mechanism, ask "which adversary does this stop, and which does it not touch?" That one question is worth more than any single technique.
What do I do, and what does each option cost?
Can I see it myself?
Measure the cost of encryption on columnar scans
A 1% range query on 20 million Parquet rows: sorted ts, ts as a random permutation, and ts encrypted with AES and decrypted at query time.
MeasurePython with PyArrow, DuckDB and cryptographyEnvelope encryption and crypto-shredding in 40 lines
Encrypt 5 patients with their own data keys wrapped by a master key, time 100 MB of AES-GCM, shred one patient, then restore a backup.
MeasurePython with cryptographyShow that a masked column is still searchable
Recover the three hidden digits of a masked 6-digit ID with prefix count queries, then reject predicates on the masked column.
Break itPython (SQLite)Crack a hashed phone number in seconds
Hash every 7-digit number with SHA-256, recover a hashed number by lookup, then try again with a per-record salt.
Break itPythonRe-identify a "5-anonymous" table and measure the uniqueness of quasi-identifiers
100,000 fake people: how many are unique on birth date, postcode and sex, and what k two 5-anonymous releases reach once joined.
Break itPython with pandasDifferencing attack and the Laplace mechanism
Recover one salary from two exact sums, then add Laplace noise to counts and measure the error at true counts 5, 100 and 1,000,000.
Break itPython with NumPyA privacy budget ledger
A noisy count that charges epsilon to a budget of 1.0: how many of 15 queries are answered, and what averaging repeats costs.
SimulatePython with NumPyA result cache that leaks, and the fix
A result cache keyed only by query text serves north rows to a south user; keying by identity and policy fixes it and removes the timing signal.
Break itPython
Can I reason about it under questioning?
Interview questions
Short questions on security and privacy in data platforms, such as identity and authorization, where access is enforced, encryption and tokenization, anonymization and differential privacy, and where controls break, each with an answer you could give aloud and a link to the full explanation.
Scenarios
Longer problems on security and privacy in data platforms, of the kind asked in senior interviews: each one estimated with numbers, diagnosed, and answered with its trade-offs.
The tensions this mechanism takes a position on. Each recurs across mechanisms.
Ideas this mechanism meets that keep coming back, mechanism after mechanism.
Guarantees have scope
Atomicity is bounded by the shared commit substrate.
Absence is not evidence
Absence of an event is not absence of a failure.
Costs are conserved
Costs are conserved, never removed.
The algebra decides
What can be skipped, pushed down, parallelized, merged or precomputed is determined by the algebra of the function, not the layout of the data.
Fixed capacity meets growth
A fixed-capacity structure meeting growing cardinality breaks the design.