Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Data minimization and the right to be forgotten

Why is minimization the only free privacy control, and why is erasure a reachability problem over lineage?

Data minimization and purpose limitation

Data minimization means collecting only what you need. means that data collected for purpose X may be used only for X.

This is the one mechanism here with no utility-versus-privacy trade-off at the point of application, because data that was never collected cannot leak, cannot be demanded by a court, cannot be re-identified, and costs nothing to delete. Every other mechanism is a mitigation for a decision that was already made.

It is also the mechanism most opposed to how data platforms are built. The modern stack rests on three beliefs: storage is cheap, so collect everything; (store raw data now, decide its meaning when you query it), so decide later; and the value of data is in joins you have not thought of yet. Minimization negates all three. Purpose limitation negates the lakehouse's central promise that any dataset can answer any future question.

So minimization is an organizational constraint, not a technical one, and it fails for the reason organizational constraints fail. The engineer who adds a column ships a feature today. The exposure is paid later, by someone else, possibly another company. This is the law that : the party blocked pays now and the party exposed pays later. Minimization has no natural forcing function at all: no error, no extra latency, no bill. Regulation exists exactly because there is no signal inside the system.

Where it is implemented, the tools are ones you already know, pointed at a new goal:

  • required purpose tags on columns;
  • retention as a schema property (each column carries "delete after N days");
  • collection-time filtering instead of ingest-everything-then-filter;
  • derived-only storage: keep the aggregate and discard the events.

The last one is the most under-used and has the highest leverage. Most pipelines keep raw event-level data forever, to serve aggregates that could have been computed once.

Right to be forgotten

The (also called the right to erasure, a right in GDPR and similar laws) lets a person ask an organization to delete their data.

The mechanics are the law we already know: immutable storage grows an indirection layer plus a collector. Parquet files and table snapshots are never edited in place, so a delete becomes a note ("this row is deleted") plus a later cleanup job that rewrites the files. The escape (destroy the key instead of the data) was covered in the Protection article. What is new here is scope, and scope is where erasure really fails.

over the lineage graph (the map of which data came from which). The person's data exists in:

  • the raw table;
  • every derived table, aggregate and materialized view;
  • every cache and replica;
  • every snapshot and backup;
  • every extract that someone took;
  • every downstream system fed by (pushing warehouse data back into operational tools);
  • every log line;
  • every model trained on the data.

A deletion that touches only the source is theatre. Wei's 14 tables, caches, CSV files and the model are exactly this list.

There are three hard sub-problems, in increasing order of how unsolved they are.

Aggregates. If an aggregate includes the person, does deletion require recomputing it? Recompute everything and the cost has no bound. Recompute nothing and the person's value survives in derived form. Worse, an aggregate published before and after a deletion is a . Erasure itself leaks: the change between the two versions reveals the deleted person's contribution. This is an uncomfortable result with no clean answer.

Models. Trained models memorize, and this is demonstrated for outliers and rare sequences. Extraction attacks against models are practical. (making a model forget some training data) is an active research area with no cheap general solution. Retraining without the data is the only sound method, and it is too expensive at any regular cadence. This is currently unsolved, not just hard. Anyone who says otherwise is describing an approximation.

Backups. Immutable backups are a durability requirement and, at the same time, a deletion obstacle. The usual answer is to delete on restore (re-apply the deletion list whenever a backup is restored) and not inside the backup. That is a procedural control, not a technical guarantee. It holds only as long as people follow the procedure.

The structural point: deletion propagation needs column-level lineage that is complete. Real lineage (the record of which columns feed which tables, kept by the catalog) is usually partial, best-effort, and parsed from SQL text and not observed at run time. A deletion guarantee inherits the coverage of your lineage graph. This is the tightest real link between the metadata and the privacy parts of the platform.