Labs

Measure the per-file overhead

The same data as 800 files of 0.5 MB, 8 files and 1 file, scanned by DuckDB from local disk and from an S3-compatible store; F and b fitted.

Z-order versus sort: count the files touched

10,000 files over four columns, clustered by a sort on A and by a Z-order on A–D: files read by a filter on A and on D.

Overlap depth before and after a blind compaction

200 small files with narrow key ranges in arrival order; overlap depth before and after compacting by time and by key range.

Retention window versus storage

A model of a table format: 10% rewritten daily for 10 days, storage after expiry and vacuum for 1-, 3- and 7-day windows; then a long query after a short vacuum.

Find the square-root interval by simulation

A read penalty that grows by r per hour since the last compaction and a compaction cost C: the cheapest interval against √(2C/r), then with 4r.

Find the compaction interval

Measure the scan cost per unmerged delta file and the cost of a rewrite, compute the compaction interval τ*, and compare a day at τ*/4, τ* and 4τ*.

Verify a rewrite

Compact 50 Parquet files into one sorted file; compare count and an order-independent fingerprint; then drop a row and replace a row with a duplicate.