Verification of the rewrite

How do you prove a maintenance rewrite preserved the data, almost for free?

Compaction already reads every byte, so computing a row count and a per-column hash aggregate before and after costs almost nothing. Yet almost nobody does it. A silent corruption caused by a maintenance rewrite has no user-visible trigger and no failing job.

-- Run on the input files, then on the output files of the rewrite:
SELECT count(*)                         AS row_count,
       sum(hash(event_id, status, ts)) AS content_fingerprint
FROM   events_partition_input;          -- same query on the output; results must match

The two results must be equal. (A sum of per-row hashes does not depend on row order, so it survives re-sorting. Overflow behavior depends on the engine.) This is write-audit-publish applied to maintenance: write the new files, audit them, and only then publish them with a commit.

Case, resolvedConstructed case · Nadia at Stackwise

A compaction that fails every night

Compaction of the busiest partition, with 12 writers appending every minute, failed every night. Where it started: Running maintenance safely →

The failing compaction. The nightly job tried to rewrite 14,000 files in one 40-minute transaction while 12 writers committed every minute. Under the coarse rule, any commit during those 40 minutes made it fail. She switches to narrow conflict detection and splits the work into small jobs of about 500 files (a few minutes each), each of which rebases over new appends. The partition now compacts every hour.

Trigger. She stops using a nightly schedule. She measures C and r for the busy partition, finds T of about 3 hours, sets cost-based triggers per partition, and keeps a threshold alarm on file count as a safety net.

Verification. Every compaction commit now includes the row count and fingerprint check. One week later it catches a bug in a new writer (not in compaction) that duplicated rows. The check pays for itself.