Concurrency: maintenance competes with writers

Why does a long compaction keep losing to short appends, and how does semantic conflict detection fix it?

A compaction is a long transaction. It competes through the optimistic CAS on the manifest with short appends from writers. The probability of a conflict rises with the length of the rewrite window, and that window rises with bytes. So large compactions on hot tables starve permanently. This is OCC livelock (everyone keeps working but one party never finishes), the same failure as in an ordinary optimistic database, except here the job duration is minutes instead of microseconds.

The escape: semantic conflict detection. A compaction is logically a no-op: it changes no answer. So it commutes (can be reordered) with any commit that only adds files. It truly conflicts only with commits that removed or modified the files it consumed.

Small example. Compaction reads files F1-F100 and writes G1. While it runs, a writer adds F101. With a coarse rule ("the table changed, so fail"), compaction fails. With the narrow rule ("did anyone touch F1-F100?"), compaction can be rebased: its commit says "remove F1-F100, add G1" on top of the new state that already contains F101. It succeeds. If a concurrent delete had rewritten F37, then it truly conflicts, and the compaction must abort or redo that part.

Narrowing the conflict predicate from "the table changed" to "my input file set was touched" is what makes background maintenance possible at all.