Find the square-root interval by simulation
Simulate a modelNeeds Python
Problem
Between compactions, a table accumulates small files and deletes that make every read a little slower, so the read penalty grows with the time since the last compaction; compacting resets it at a fixed cost. Compacting rarely saves compaction cost and pays in reads; compacting often does the reverse. With a penalty growing at r per hour and a compaction cost C, the cost per hour is C/T + rT/2, minimal at T* = √(2C/r).
Model a table whose extra read cost grows by r per hour each hour since the last compaction, with a compaction cost of C = 1,800 and r = 0.25. Simulate 1,000 hours for intervals T from 1 to 500 hours and plot the total cost per hour. Then quadruple r.
The question. Where is the cheapest compaction interval, and how does it move when r quadruples?
Predict
Expect a minimum near 120 hours. Quadrupling r should halve the best T.
Run it
1Install uv once. It fetches Python and the lab's packages for you.
Terminal curl -LsSf https://astral.sh/uv/install.sh | sh2Get the script. It is the whole lab: read it, then download it.
View the codeHide the code
find-the-square-root-interval-by.py # /// script # requires-python = ">=3.10" # dependencies = ["numpy"] # /// # Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT # https://thedataengineering.com/lab-code/LICENSE.txt """Find the square-root interval by simulation. A table whose extra read cost grows by r per hour for every hour since the last compaction, and a compaction that costs C. Simulates 1,000 hours for intervals T from 1 to 500 hours, finds the cheapest, and compares with √(2C/r); then quadruples r. """ import argparse, json import numpy as np p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) p.add_argument("--c", type=float, default=1800, help="Compaction cost C.") p.add_argument("--r", type=float, default=0.25, help="Read-cost growth per hour r.") p.add_argument("--hours", type=int, default=1000, help="Simulated hours.") p.add_argument("--json", help="Also write the results to this JSON file.") args = p.parse_args() def simulate(C, r): out = [] for T in range(1, 500): cost, age = 0.0, 0 for h in range(args.hours): cost += r * age; age += 1 # the read penalty grows with age if age == T: cost += C; age = 0 # compact out.append({"T": T, "cost_per_hour": cost / args.hours, "formula": C / T + r * T / 2}) return out rows = [] for r in (args.r, 4 * args.r): sim = simulate(args.c, r); best = min(sim, key=lambda x: x["cost_per_hour"]) rows.append({"r": r, "best_T": best["T"], "formula_T": (2 * args.c / r) ** 0.5, "best_cost": best["cost_per_hour"]}) if r == args.r: curve = [x for x in sim if x["T"] % 5 == 0 or x["T"] < 5] for x in rows: print(f"r = {x['r']}: best T = {x['best_T']} h (formula √(2C/r) = {x['formula_T']:.0f} h), cost {x['best_cost']:.1f} per hour") if args.json: with open(args.json, "w") as f: json.dump({"tables": {"best": rows, "curve": curve}}, f)3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal uv run find-the-square-root-interval-by.py uv run find-the-square-root-interval-by.py --help # every knob
| Knob · default | What it changes |
|---|---|
| --c1800 | Compaction cost C. |
| --r0.25 | Read-cost growth per hour r. |
| --hours1000 | Simulated hours. |
Our run
Data table
| Interval T (hours) | Simulated | C/T + rT/2 |
|---|---|---|
| 1 | 1,800 | 1,800 |
| 2 | 900 | 900 |
| 3 | 600 | 600 |
| 4 | 450 | 451 |
| 5 | 361 | 361 |
| 10 | 181 | 181 |
| 15 | 121 | 122 |
| 20 | 92.4 | 92.5 |
| 25 | 75 | 75.1 |
| 30 | 63 | 63.8 |
| 35 | 54.6 | 55.8 |
| 40 | 49.9 | 50 |
| 45 | 45.1 | 45.6 |
| 50 | 42.1 | 42.3 |
| 55 | 39.1 | 39.6 |
| 60 | 36.1 | 37.5 |
| 65 | 34.9 | 35.8 |
| 70 | 33.7 | 34.5 |
| 75 | 32.5 | 33.4 |
| 80 | 31.3 | 32.5 |
| 85 | 30.1 | 31.8 |
| 90 | 30.8 | 31.3 |
| 95 | 29.5 | 30.8 |
| 100 | 30.4 | 30.5 |
| 105 | 28.9 | 30.3 |
| 110 | 29.7 | 30.1 |
| 115 | 28.3 | 30 |
| 120 | 28.9 | 30 |
| 125 | 29.9 | 30 |
| 130 | 28.3 | 30.1 |
| 135 | 28.8 | 30.2 |
| 140 | 29.7 | 30.4 |
| 145 | 28.6 | 30.5 |
| 150 | 28.8 | 30.8 |
| 155 | 29.3 | 31 |
| 160 | 30.1 | 31.3 |
| 165 | 31.1 | 31.5 |
| 170 | 29.8 | 31.8 |
| 175 | 30 | 32.2 |
| 180 | 30.4 | 32.5 |
| 185 | 31 | 32.9 |
| 190 | 31.8 | 33.2 |
| 195 | 32.7 | 33.6 |
| 200 | 33.9 | 34 |
| 205 | 32.1 | 34.4 |
| 210 | 32.3 | 34.8 |
| 215 | 32.6 | 35.2 |
| 220 | 33.1 | 35.7 |
| 225 | 33.6 | 36.1 |
| 230 | 34.3 | 36.6 |
| 235 | 35.1 | 37 |
| 240 | 36.1 | 37.5 |
| 245 | 37.1 | 38 |
| 250 | 38.3 | 38.5 |
| 255 | 36.6 | 38.9 |
| 260 | 36.7 | 39.4 |
| 265 | 36.9 | 39.9 |
| 270 | 37.1 | 40.4 |
| 275 | 37.5 | 40.9 |
| 280 | 37.9 | 41.4 |
| 285 | 38.4 | 41.9 |
| 290 | 38.9 | 42.5 |
| 295 | 39.6 | 43 |
| 300 | 40.3 | 43.5 |
| 305 | 41.1 | 44 |
| 310 | 41.9 | 44.6 |
| 315 | 42.9 | 45.1 |
| 320 | 43.9 | 45.6 |
| 325 | 45 | 46.2 |
| 330 | 46.1 | 46.7 |
| 335 | 45.1 | 47.2 |
| 340 | 45.2 | 47.8 |
| 345 | 45.2 | 48.3 |
| 350 | 45.4 | 48.9 |
| 355 | 45.5 | 49.4 |
| 360 | 45.7 | 50 |
| 365 | 45.9 | 50.6 |
| 370 | 46.1 | 51.1 |
| 375 | 46.4 | 51.7 |
| 380 | 46.8 | 52.2 |
| 385 | 47.1 | 52.8 |
| 390 | 47.5 | 53.4 |
| 395 | 48 | 53.9 |
| 400 | 48.5 | 54.5 |
| 405 | 49 | 55.1 |
| 410 | 49.5 | 55.6 |
| 415 | 50.1 | 56.2 |
| 420 | 50.8 | 56.8 |
| 425 | 51.4 | 57.4 |
| 430 | 52.1 | 57.9 |
| 435 | 52.9 | 58.5 |
| 440 | 53.7 | 59.1 |
| 445 | 54.5 | 59.7 |
| 450 | 55.4 | 60.3 |
| 455 | 56.2 | 60.8 |
| 460 | 57.2 | 61.4 |
| 465 | 58.1 | 62 |
| 470 | 59.1 | 62.6 |
| 475 | 60.2 | 63.2 |
| 480 | 61.3 | 63.8 |
| 485 | 62.4 | 64.3 |
| 490 | 63.5 | 64.9 |
| 495 | 64.7 | 65.5 |
Best interval
| r | Best T (h) | √(2C/r) (h) | Cost per hour |
|---|---|---|---|
| 0.25 | 126 | 120 | 28.1 |
| 1 | 59 | 60 | 57.7 |
r = 0.25: best T = 126 h (formula √(2C/r) = 120 h), cost 28.1 per hour
r = 1.0: best T = 59 h (formula √(2C/r) = 60 h), cost 57.7 per hourWhat we found
The cheapest interval was 126 hours against √(2C/r) = 120; with r quadrupled it was 59 hours against 60, half. The curve is flat around its minimum, so the interval can be off by a fair margin at little cost.
Read it
The square-root rule says how compaction should respond to change: four times the write rate (and so the read penalty's growth) halves the interval, it does not quarter it. Measure r and C on your table, and adjust the schedule when the write rate moves.