Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Protection gaps and residency

The protection gaps

Integrity and authenticity. The Protection article was about confidentiality. The other half of the property space is has this been modified? For a data platform this is arguably the higher risk, because the highest-leverage target is not a column of personal data. It is the metadata. The table manifests and snapshot trees decide which files make up a table. Changing one line of metadata silently changes every query result, leaves the data files untouched, and is invisible to every confidentiality control.

without is the specific failure: unauthenticated ciphertext is malleable (an attacker can change it in controlled ways). That is why (authenticated encryption with associated data, which encrypts and checks integrity in one step) is the only acceptable construction. "We encrypt it" without "we authenticate it" is an incomplete sentence.

Tamper-evident logs are the matching mechanism for audit. Use : each log entry includes the of the previous entry, so editing any old entry breaks every hash after it. The reason is simple: an audit log that an attacker can edit records only the honest actors.

Data in use. At rest and in transit are two of three data states. The third is in use. Plaintext exists in memory during every query, so the platform operator, the hypervisor and anyone with host access can read it whatever the key policy says. (running code in a hardware-protected enclave that even the host cannot read) moves the trust boundary to the hardware and its (a signed proof of what code is running). It is the only mechanism that answers the platform-operator without you owning all the compute. Today: the performance overhead is modest, side-channel attacks on these environments are an active research area, and the guarantee rests on trusting the silicon vendor's attestation root. That is trust relocated, not removed (costs are relocated, never removed).

Nonce and randomness management. A is a number that must be used only once with a given key. This is the most common catastrophic implementation failure in practice, and it is not a design choice. It is an invariant. Reusing a nonce under the same key with a stream or counter-mode cipher turns a strong construction into a trivially broken one, including loss of authentication. It matters for data platforms in particular because parallel writers and retried tasks are the two situations most likely to produce a collision, and both are structural to how this stack works. If a task retry re-encrypts the same block with the same derived nonce, the design was wrong before anyone attacked it. A safe pattern: derive the nonce from a unique task-attempt ID, or use random 96-bit nonces with a bounded number of messages per key, or use a nonce-misuse-resistant mode.

Custody and compulsion. One question decides most of this topic: is the infrastructure provider inside your ?

  • If yes, provider-managed keys give you auditability and nothing else. The party that holds the key can be forced by law, breached, or abused by an insider.
  • Customer-managed keys in the provider's key service narrow the exposure.
  • Hold-your-own-key and client-side encryption narrow it further, but you lose server-side processing completely, because the provider cannot query what it cannot decrypt.

That is the real trade: over the same bytes, unless you use confidential computing.

Pipeline secrets. These are different from and are more often the actual leak path: connection strings, API tokens, and the a pipeline needs to run. They end up in code, configuration files, orchestrator variables, environment dumps and logs. The fix is the same as everywhere else here: replace the stored secret with a short-lived credential obtained through . Its absence is why credentials in code repositories remain the most reliably productive attack in the field. (Wei's pasted token in Identity and access was this case.)

Residency and sovereignty

Data residency is a hard placement rule from outside the system: certain data may not physically leave a country or region. It overrides the platform's choices about where replicas go and about the cost of moving data between regions, and none of the controls described so far can satisfy it, because the rule is about location, not about access.

It is the clearest case of an outside constraint reaching several layers down and removing options that were best on every internal measure. It also gives a conclusion that is uncomfortable but correct: encryption does not solve residency. The objection is usually to the legal reachability of the data by a foreign authority, not to whether it can be read. A provider that is subject to a foreign legal system can be forced to act whatever the location of the bytes. That is why sovereignty requirements keep ending up as demands for operational independence, not for cryptography.