Model
Deduplication window
Every dedup window in production is a bet that the retry arrives before you forget the key.
- Window
- how long keys are remembered
How it behaves
One input on the horizontal axis, the others held fixed.
Data table
| Window W | Key storage |
|---|---|
| 60 s | 183 MB |
| 67.3 s | 205 MB |
| 75.6 s | 231 MB |
| 84.8 s | 259 MB |
| 95.1 s | 290 MB |
| 107 s | 326 MB |
| 120 s | 366 MB |
| 134 s | 410 MB |
| 151 s | 460 MB |
| 169 s | 517 MB |
| 190 s | 580 MB |
| 213 s | 650 MB |
| 239 s | 730 MB |
| 268 s | 819 MB |
| 301 s | 919 MB |
| 338 s | 1.01 GB |
| 379 s | 1.13 GB |
| 425 s | 1.27 GB |
| 477 s | 1.42 GB |
| 536 s | 1.6 GB |
| 601 s | 1.79 GB |
| 675 s | 2.01 GB |
| 757 s | 2.26 GB |
| 849 s | 2.53 GB |
| 953 s | 2.84 GB |
| 1,070 s | 3.19 GB |
| 1,200 s | 3.58 GB |
| 1,350 s | 4.01 GB |
| 1,510 s | 4.5 GB |
| 1,700 s | 5.05 GB |
| 1,900 s | 5.67 GB |
| 2,140 s | 6.36 GB |
| 2,400 s | 7.14 GB |
| 2,690 s | 8.01 GB |
| 3,020 s | 8.99 GB |
| 3,390 s | 10.1 GB |
| 3,800 s | 11.3 GB |
| 4,260 s | 12.7 GB |
| 4,780 s | 14.3 GB |
| 5,370 s | 16 GB |
| 6,020 s | 18 GB |
| 6,760 s | 20.1 GB |
| 7,590 s | 22.6 GB |
| 8,510 s | 25.4 GB |
| 9,550 s | 28.5 GB |
| 10,700 s | 31.9 GB |
| 12,000 s | 35.8 GB |
| 13,500 s | 40.2 GB |
| 15,100 s | 45.1 GB |
| 17,000 s | 50.6 GB |
| 19,100 s | 56.8 GB |
| 21,400 s | 63.8 GB |
| 24,000 s | 71.6 GB |
| 26,900 s | 80.3 GB |
| 30,200 s | 90.1 GB |
| 33,900 s | 101 GB |
| 38,100 s | 113 GB |
| 42,700 s | 127 GB |
| 47,900 s | 143 GB |
| 53,800 s | 160 GB |
| 60,400 s | 180 GB |
| 67,700 s | 202 GB |
| 76,000 s | 227 GB |
| 85,300 s | 254 GB |
| 95,700 s | 285 GB |
| 107,000 s | 320 GB |
| 121,000 s | 359 GB |
| 135,000 s | 403 GB |
| 152,000 s | 452 GB |
| 170,000 s | 507 GB |
| 191,000 s | 569 GB |
| 214,000 s | 639 GB |
| 241,000 s | 717 GB |
| 270,000 s | 805 GB |
| 303,000 s | 903 GB |
| 340,000 s | 1,010 GB |
| 381,000 s | 1.11 TB |
| 428,000 s | 1.25 TB |
| 480,000 s | 1.4 TB |
| 539,000 s | 1.57 TB |
| 605,000 s | 1.76 TB |
Worked examples, recomputed
Every number the mechanisms quote for this relationship is recomputed from it, so none of them disagrees with it.
Where it appears
Each mechanism states it in its own names. The general symbol each name stands for is shown with it.
- Fault Tolerance & Delivery Semantics
What is the cheapest thing that can be safely done twice?