Skip to content
About
Main content

Find the square-root interval by simulation

Simulate a modelNeeds Python

Problem

Between , a table accumulates small files and deletes that make every read a little slower, so the read penalty grows with the time since the last compaction; compacting resets it at a fixed cost. Compacting rarely saves compaction cost and pays in reads; compacting often does the reverse. With a penalty growing at r per hour and a compaction cost C, the cost per hour is C/T + rT/2, minimal at T* = √(2C/r).

Model a table whose extra read cost grows by r per hour each hour since the last compaction, with a compaction cost of C = 1,800 and r = 0.25. Simulate 1,000 hours for intervals T from 1 to 500 hours and plot the total cost per hour. Then quadruple r.

The question. Where is the cheapest compaction interval, and how does it move when r quadruples?


Predict

Quadrupling the rate r: what happens to the cheapest compaction interval?

Run it

  1. 1Install uv once. It fetches Python and the lab's packages for you.

    Terminal
    curl -LsSf https://astral.sh/uv/install.sh | sh
  2. 2Get the script. It is the whole lab: read it, then download it.

    find-the-square-root-interval-by.py43 linesDownload
    View the code
    find-the-square-root-interval-by.py
    # /// script
    # requires-python = ">=3.10"
    # dependencies = ["numpy"]
    # ///
    # Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
    # https://thedataengineering.com/lab-code/LICENSE.txt
    """Find the square-root interval by simulation.
    
    A table whose extra read cost grows by r per hour for every hour since the
    last compaction, and a compaction that costs C. Simulates 1,000 hours for
    intervals T from 1 to 500 hours, finds the cheapest, and compares with
    √(2C/r); then quadruples r.
    """
    import argparse, json
    import numpy as np
    
    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    p.add_argument("--c", type=float, default=1800, help="Compaction cost C.")
    p.add_argument("--r", type=float, default=0.25, help="Read-cost growth per hour r.")
    p.add_argument("--hours", type=int, default=1000, help="Simulated hours.")
    p.add_argument("--json", help="Also write the results to this JSON file.")
    args = p.parse_args()
    
    def simulate(C, r):
        out = []
        for T in range(1, 500):
            cost, age = 0.0, 0
            for h in range(args.hours):
                cost += r * age; age += 1                                          # the read penalty grows with age
                if age == T: cost += C; age = 0                                    # compact
            out.append({"T": T, "cost_per_hour": cost / args.hours, "formula": C / T + r * T / 2})
        return out
    rows = []
    for r in (args.r, 4 * args.r):
        sim = simulate(args.c, r); best = min(sim, key=lambda x: x["cost_per_hour"])
        rows.append({"r": r, "best_T": best["T"], "formula_T": (2 * args.c / r) ** 0.5, "best_cost": best["cost_per_hour"]})
        if r == args.r: curve = [x for x in sim if x["T"] % 5 == 0 or x["T"] < 5]
    for x in rows: print(f"r = {x['r']}: best T = {x['best_T']} h (formula √(2C/r) = {x['formula_T']:.0f} h), cost {x['best_cost']:.1f} per hour")
    
    if args.json:
        with open(args.json, "w") as f:
            json.dump({"tables": {"best": rows, "curve": curve}}, f)
    
  3. 3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.

    Terminal
    uv run find-the-square-root-interval-by.py
    uv run find-the-square-root-interval-by.py --help     # every knob
Knobs
Knob · defaultWhat it changes
--c1800 Compaction cost C.
--r0.25 Read-cost growth per hour r.
--hours1000 Simulated hours.

Our run

Cost per hour against the compaction interval (r = 0.25)Apple M2 · 8 cores · 8 GB
SimulatedC/T + rT/2
Data table
Cost per hour against the compaction interval (r = 0.25)
Interval T (hours)SimulatedC/T + rT/2
11,8001,800
2900900
3600600
4450451
5361361
10181181
15121122
2092.492.5
257575.1
306363.8
3554.655.8
4049.950
4545.145.6
5042.142.3
5539.139.6
6036.137.5
6534.935.8
7033.734.5
7532.533.4
8031.332.5
8530.131.8
9030.831.3
9529.530.8
10030.430.5
10528.930.3
11029.730.1
11528.330
12028.930
12529.930
13028.330.1
13528.830.2
14029.730.4
14528.630.5
15028.830.8
15529.331
16030.131.3
16531.131.5
17029.831.8
1753032.2
18030.432.5
1853132.9
19031.833.2
19532.733.6
20033.934
20532.134.4
21032.334.8
21532.635.2
22033.135.7
22533.636.1
23034.336.6
23535.137
24036.137.5
24537.138
25038.338.5
25536.638.9
26036.739.4
26536.939.9
27037.140.4
27537.540.9
28037.941.4
28538.441.9
29038.942.5
29539.643
30040.343.5
30541.144
31041.944.6
31542.945.1
32043.945.6
3254546.2
33046.146.7
33545.147.2
34045.247.8
34545.248.3
35045.448.9
35545.549.4
36045.750
36545.950.6
37046.151.1
37546.451.7
38046.852.2
38547.152.8
39047.553.4
3954853.9
40048.554.5
4054955.1
41049.555.6
41550.156.2
42050.856.8
42551.457.4
43052.157.9
43552.958.5
44053.759.1
44554.559.7
45055.460.3
45556.260.8
46057.261.4
46558.162
47059.162.6
47560.263.2
48061.363.8
48562.464.3
49063.564.9
49564.765.5
macOS 26.6.2 · Python 3.13.3 · numpy 2.5.4 · run 2026-10-20

Best interval

Best interval
rBest T (h)√(2C/r) (h)Cost per hour
0.2512612028.1
1596057.7
Output · uv run find-the-square-root-interval-by.py
r = 0.25: best T = 126 h (formula √(2C/r) = 120 h), cost 28.1 per hour
r = 1.0: best T = 59 h (formula √(2C/r) = 60 h), cost 57.7 per hour

What we found

The cheapest interval was 126 hours against √(2C/r) = 120; with r quadrupled it was 59 hours against 60, half. The curve is flat around its minimum, so the interval can be off by a fair margin at little cost.


Read it

The square-root rule says how compaction should respond to change: four times the write rate (and so the read penalty's growth) halves the interval, it does not quarter it. Measure r and C on your table, and adjust the schedule when the write rate moves.