Delete compaction and orphan detection

When is it worth rewriting a file to drop deleted rows, and how do you find orphans safely?

Delete (tombstone) compaction

A file in which a fraction d of rows are deleted (by delete vectors) still costs a full read, plus the work of applying the deletes. Rewriting it removes that overhead. A simple rule: rewrite when

(in units of "one file read": the rewrite costs about one read of the file, and each future read saves about d of one read).

Example: a file with d = 0.2 deleted rows and 50 expected reads: 0.2 × 50 = 10 > 1, so rewrite. A file with d = 0.2 and 3 expected reads: 0.6 < 1, so leave it.

A rewrite also moves rows, which invalidates everything that refers to a position: positional delete vectors ("row 17 of this file") and row-position indexes. So compaction must merge delete vectors into the new file rather than carry them across, and the two jobs cannot be scheduled independently. More generally, every rewrite leaves derived state (statistics, indexes) describing a layout that no longer exists, and the job that rewrites must say who refreshes it.

This rule needs to know the read frequency per file, which lives in usage telemetry. Most teams do not have per-file access data, so everyone falls back to fixed thresholds ("rewrite when 10% of rows are deleted"). Maintenance without usage data is guesswork, and that is the honest state of practice.

---

Orphan detection (the operational view)

Reclamation explained why it needs a full LIST and an anti-join and why the safe window is measured in days. Operationally this means: run it rarely, run it with a long grace period (for example, only files older than 3 days), never run it while a known long writer is active, and log every deleted path so a mistake can be investigated.