Reconciliation and drift

In which order must writes to two stores happen, and how do you detect the rest?

Two stores that cannot be updated atomically will diverge. Divergence goes in two directions with very different costs:

Situation	Name	Effect
Files exist but no metadata points to them	orphans	waste money; threaten nothing
Metadata points to files that do not exist	dangling references	queries fail hard

That asymmetry dictates the order of writes:

On write: data first, then metadata.
On delete: metadata first, then data.
Never the reverse.

A crash between the two writes

If a crash happens between the two steps, the first order leaves an orphan (harmless) and the second never leaves a dangling reference. This is the same discipline as the write-ahead log (a database writes its intention to a log before changing the data pages) and the transactional outbox (a service stores "message to send" in the same database transaction as its change, and sends later). The general rule:

Detection is expensive. A reconciliation scan is O(files) LIST operations, each with a per-request price. So how often to scan is an economic decision.

Orphan deletion needs an age threshold longer than the longest possible in-flight write, or you delete files that a slow writer is about to commit. That threshold is an assumption the system cannot verify. When it fails, the one who pays is the writer whose eight-hour job has its output collected out from under it. The same shape returns with usage-based deprecation.

Order cross-store writes so the survivable inconsistency is the one that occurs. Detect the rest at a cadence you can afford.

Case, resolvedConstructed case · Hana at Trailmix

A rename, and the table's history vanished

Renaming orders to sales_orders emptied its lineage, a cleanup bot proposed deleting it for zero usage, an hourly events table spent 40 seconds planning, and a streaming job hit commit errors. Where it started: The addressing layer →

Rename problem. The catalog keyed everything on the name orders. She asks the platform team to key lineage, ownership and usage on a stable asset ID and to treat the name as an attribute. After the fix, a rename keeps its history, and the cleanup bot sees the real usage and drops the proposal.
40-second planning. The events table has about 26,000 hourly partitions, each with a few small files, in a single relational metastore. She changes the writer to produce larger files, switches to daily partitions (about 1,100 instead of 26,000), and moves the table to a manifest-based format, so pruning skips whole manifests. Planning drops to about two seconds.
Commit errors. Two writers raced on the pointer, and one swap was rejected. The streaming job had no retry-and-revalidate step. She adds it, and batches commits to one every 10 seconds, so that a few commit round trips no longer limit it.

She also notes the order rule for the weekly cleanup: delete metadata first, data second, and never delete files younger than 24 hours (longer than any job she knows about).