Privacy is a bound on what an adversary learns

Why is the only privacy lever the destruction of information?

CaseConstructed case · Dr. Adaeze at Hollis Health

A 5-anonymous extract with a named politician in it

Dr. Adaeze Eze, Hollis's privacy officer, wants to publish a "research extract" of the visits table so that university partners can study hospital readmissions. Wei removes the names and the national_id, and a tool reports that the extract is "5-anonymous". Dr. Adaeze signs it off.

Three weeks later, three things happen in one week:

A journalist tells Hollis she found a named local politician in the extract. She used a public news story ("admitted on a Tuesday in March") and the politician's postcode and birth year.
A second extract, built for a different research group with different generalizations, is published. The same journalist combines the two and finds three more patients.
A patient sends an erasure request under the data-protection law: "delete everything you hold about me." Wei finds the patient's data in 14 tables, 3 dashboards' caches, nightly backups, two exported CSV files, and a readmission-risk model that was trained last quarter.

Nothing was hacked. Every access was authorized and every byte was decrypted by someone who was allowed to decrypt it. The leaks came from outputs the system was built to produce. This article explains why that is a different kind of problem, and what can and cannot be done about it.

How it ends →

The two previous articles assumed a party who should not have the data. Access control denies that party the request. Encryption denies that party the bytes. Privacy mechanisms exist for the case where there is no such party: every access was authorized, every byte was legitimately decrypted, and the disclosure happened anyway, through an output that the system was built to produce.

That changes the character of the constraint.

Access control failures are engineering failures: a wrong grant, a missed chokepoint.
Encryption failures are engineering failures too.
Privacy failures are not failures in the same sense. The system worked. The leak is a property of what the data means, carried through an operation that looks exactly like the operation you wanted.

Two things follow.

First, the adversary's power now includes knowledge you do not control. This knowledge includes auxiliary datasets (other data the adversary already holds, such as a voter list or a news article), background facts about the target, and the ability to ask more than one question. You cannot list all this knowledge, so you cannot list all the attacks.

Second, and this is the hard part, the remedy is not a better algorithm. Confidentiality was computational: make the attacker's work expensive and you win. Privacy is informational: if the information is present in the output, it is disclosed, and no amount of computation on the defending side changes that. This is the first constraint in the whole platform that engineering effort cannot improve away.

So the only lever left is destruction of information: generalize, perturb, aggregate, or never collect. Each of these lowers the information content of the release. That lowers disclosure and utility together, because they are the same quantity.

This sentence compresses the history of the field. The first generation of mechanisms tried to define privacy as a property of the released table. Twenty years of attacks showed this cannot work. The second generation redefined privacy as a property of the release process.