Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Manifest rewriting and orphan files

Why do small files cost twice through planning, and how do files nobody references appear?

Manifest rewriting: metadata has the same disease

Metadata is a table too, and it has the same pathology: it is append-only, there is one per , and a must read all of it before the query can start.

Planning cost scales with the number of files, which scales as 1 / file size. : once on the data path (the fixed cost per file, Layout restoration), and once on the metadata path. The second bill is the serious one.

Planning is largely serial and sits in front of all parallel work. This is Amdahl's law: the speedup of a job is limited by its serial fraction, because no number of workers can shorten the part that cannot be split. Illustrative example: at 10 million files and a planner that handles 50,000 file entries per second, planning takes 10,000,000 / 50,000 = 200 seconds (about 3.3 minutes). The rest of the query might take 600 seconds on one worker and 6 seconds on 100 workers. The total is then 206 seconds, and adding more workers helps almost nothing, because 200 of those seconds are planning.

That, and not GET pricing, is why small files are fatal

The fix for metadata is the same as for data: compact it (merge many manifests into fewer). But metadata is itself a commit, so it can conflict with writers. The recursion goes one level and stops.

Orphan files: the opposite problem

removes files that metadata no longer references. The opposite failure also happens: files on storage that metadata never referenced. A writer uploaded data files, then crashed before its commit, or lost a race and gave up. Those are invisible to the table forever. (Writing data before metadata is what makes this the harmless direction.)

Orphan detection is the one maintenance operation that cannot be driven from metadata. It needs a full LIST of the storage prefix (a call that returns the names of all objects under a path), then an against the set of known files (keep the names that are not in the known set). It is expensive and rate-limited, and the result is not a consistent : a file created in the middle of the scan looks orphaned, because the scan did not yet know about the commit that is about to reference it. That is why safe orphan windows are measured in days.