Retained-window memory

Simulate a modelNeeds Python

Problem

Allowed lateness keeps a window's state after its end so that late records can still update it. A window of size W kept for L after its end overlaps with the windows that open during that time, so a key holds about 1 + L/W windows at once. With small windows and generous lateness, that multiplies state.

For a window size W and allowed lateness L, count the windows held at once per key and check it against 1 + L/W. Try W = 1 min with L = 10 and 20 min, and W = 60 min with L = 10 min, over a simulated day.

The question. How many windows does each key hold at once for each W and L?

Predict

With 60-minute windows and 10 minutes of allowed lateness, how many windows does a key hold at once on average?

Exactly 1Just over 12 or more

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

retained-window-memory.py34 linesDownload

View the code

retained-window-memory.py

# /// script
# requires-python = ">=3.10"
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Retained window memory.

Tumbling windows of size W kept open until the watermark passes their end plus
an allowed lateness L. Counts the windows held at once for one key, as event
time advances minute by minute, against 1 + L/W.
"""
import argparse, json

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--cases", default="1:10,1:20,60:10", help="W:L pairs in minutes.")
p.add_argument("--hours", type=int, default=24, help="Event time simulated (hours).")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()

rows = []
for case in args.cases.split(","):
    W, L = map(int, case.split(":")); held = []
    for t in range(args.hours * 60 * 60):                                # one tick per second of event time
        wm = t / 60                                                       # watermark, in minutes
        first_open = int(max(0, wm - L) // W)                             # windows whose end + L is still ahead of the watermark
        last = int(wm // W)
        held.append(last - first_open + 1)
    rows.append({"W_min": W, "L_min": L, "mean_held": sum(held) / len(held), "max_held": max(held), "formula": 1 + L / W})
for r in rows: print(f"W = {r['W_min']:>2} min, L = {r['L_min']:>2} min: windows held on average {r['mean_held']:6.2f}, at most {r['max_held']}   (1 + L/W = {r['formula']:.2f})")

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"cases": rows}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal
```
uv run retained-window-memory.py
uv run retained-window-memory.py --help     # every knob
```

Knobs
Knob · default	What it changes
--cases1:10, 1:20, 60:10	W:L pairs in minutes.
--hours24	Event time simulated (hours).

Our run

Windows held per key

Windows held per key
W (min)	L (min)	On average	At most	1 + L/W
1	10	11	11	11
1	20	20.9	21	21
60	10	1.16	2	1.17

Reference runApple M2 · 8 cores · 8 GB

Output · uv run retained-window-memory.py

W =  1 min, L = 10 min: windows held on average  10.96, at most 11   (1 + L/W = 11.00)
W =  1 min, L = 20 min: windows held on average  20.85, at most 21   (1 + L/W = 21.00)
W = 60 min, L = 10 min: windows held on average   1.16, at most 2   (1 + L/W = 1.17)

macOS 26.6.2 · Python 3.13.3 · run 2026-10-20

What we found

On average a key held 11, 20.9 and 1.16 windows, against 11, 21 and 1.17 from the formula.

Read it

Allowed lateness is priced in windows held, and the price depends on its ratio to the window size: ten minutes of lateness costs almost nothing for hourly windows and multiplies state elevenfold for one-minute windows.