Model

Retry amplification across layers

Retries, timeouts and batching multiply.

A = a^{L}, T_{outer} > a \cdot T_{inner}

$a$: Attempts per layer; attempts each layer makes for one call
$L$: Layers; layers that retry
$A$: Attempts; attempts one logical request becomes
$T$: Timeouts; each outer timeout strictly larger than the inner timeout times the number of attempts

Recurring principle: Local optimizations compose badly

How it behaves

One input on the horizontal axis, the others held fixed.

2 attempts per layer3 attempts per layer

Data table

Attempts one logical request becomes, by retrying layers
Layers that retry L	series	Attempts
1	2 attempts per layer	2
1.06	2 attempts per layer	2.09
1.13	2 attempts per layer	2.18
1.19	2 attempts per layer	2.28
1.25	2 attempts per layer	2.38
1.31	2 attempts per layer	2.48
1.38	2 attempts per layer	2.59
1.44	2 attempts per layer	2.71
1.5	2 attempts per layer	2.83
1.56	2 attempts per layer	2.95
1.63	2 attempts per layer	3.08
1.69	2 attempts per layer	3.22
1.75	2 attempts per layer	3.36
1.81	2 attempts per layer	3.51
1.88	2 attempts per layer	3.67
1.94	2 attempts per layer	3.83
2	2 attempts per layer	4
2.06	2 attempts per layer	4.18
2.13	2 attempts per layer	4.36
2.19	2 attempts per layer	4.56
2.25	2 attempts per layer	4.76
2.31	2 attempts per layer	4.97
2.38	2 attempts per layer	5.19
2.44	2 attempts per layer	5.42
2.5	2 attempts per layer	5.66
2.56	2 attempts per layer	5.91
2.63	2 attempts per layer	6.17
2.69	2 attempts per layer	6.44
2.75	2 attempts per layer	6.73
2.81	2 attempts per layer	7.03
2.88	2 attempts per layer	7.34
2.94	2 attempts per layer	7.66
3	2 attempts per layer	8
3.06	2 attempts per layer	8.35
3.13	2 attempts per layer	8.72
3.19	2 attempts per layer	9.11
3.25	2 attempts per layer	9.51
3.31	2 attempts per layer	9.93
3.38	2 attempts per layer	10.4
3.44	2 attempts per layer	10.8
3.5	2 attempts per layer	11.3
3.56	2 attempts per layer	11.8
3.63	2 attempts per layer	12.3
3.69	2 attempts per layer	12.9
3.75	2 attempts per layer	13.5
3.81	2 attempts per layer	14.1
3.88	2 attempts per layer	14.7
3.94	2 attempts per layer	15.3
4	2 attempts per layer	16
4.06	2 attempts per layer	16.7
4.13	2 attempts per layer	17.4
4.19	2 attempts per layer	18.2
4.25	2 attempts per layer	19
4.31	2 attempts per layer	19.9
4.38	2 attempts per layer	20.7
4.44	2 attempts per layer	21.7
4.5	2 attempts per layer	22.6
4.56	2 attempts per layer	23.6
4.63	2 attempts per layer	24.7
4.69	2 attempts per layer	25.8
4.75	2 attempts per layer	26.9
4.81	2 attempts per layer	28.1
4.88	2 attempts per layer	29.3
4.94	2 attempts per layer	30.6
5	2 attempts per layer	32
5.06	2 attempts per layer	33.4
5.13	2 attempts per layer	34.9
5.19	2 attempts per layer	36.4
5.25	2 attempts per layer	38.1
5.31	2 attempts per layer	39.7
5.38	2 attempts per layer	41.5
5.44	2 attempts per layer	43.3
5.5	2 attempts per layer	45.3
5.56	2 attempts per layer	47.3
5.63	2 attempts per layer	49.4
5.69	2 attempts per layer	51.5
5.75	2 attempts per layer	53.8
5.81	2 attempts per layer	56.2
5.88	2 attempts per layer	58.7
5.94	2 attempts per layer	61.3
6	2 attempts per layer	64
1	3 attempts per layer	3
1.06	3 attempts per layer	3.21
1.13	3 attempts per layer	3.44
1.19	3 attempts per layer	3.69
1.25	3 attempts per layer	3.95
1.31	3 attempts per layer	4.23
1.38	3 attempts per layer	4.53
1.44	3 attempts per layer	4.85
1.5	3 attempts per layer	5.2
1.56	3 attempts per layer	5.57
1.63	3 attempts per layer	5.96
1.69	3 attempts per layer	6.38
1.75	3 attempts per layer	6.84
1.81	3 attempts per layer	7.32
1.88	3 attempts per layer	7.85
1.94	3 attempts per layer	8.4
2	3 attempts per layer	9
2.06	3 attempts per layer	9.64
2.13	3 attempts per layer	10.3
2.19	3 attempts per layer	11.1
2.25	3 attempts per layer	11.8
2.31	3 attempts per layer	12.7
2.38	3 attempts per layer	13.6
2.44	3 attempts per layer	14.6
2.5	3 attempts per layer	15.6
2.56	3 attempts per layer	16.7
2.63	3 attempts per layer	17.9
2.69	3 attempts per layer	19.2
2.75	3 attempts per layer	20.5
2.81	3 attempts per layer	22
2.88	3 attempts per layer	23.5
2.94	3 attempts per layer	25.2
3	3 attempts per layer	27
3.06	3 attempts per layer	28.9
3.13	3 attempts per layer	31
3.19	3 attempts per layer	33.2
3.25	3 attempts per layer	35.5
3.31	3 attempts per layer	38.1
3.38	3 attempts per layer	40.8
3.44	3 attempts per layer	43.7
3.5	3 attempts per layer	46.8
3.56	3 attempts per layer	50.1
3.63	3 attempts per layer	53.6
3.69	3 attempts per layer	57.5
3.75	3 attempts per layer	61.5
3.81	3 attempts per layer	65.9
3.88	3 attempts per layer	70.6
3.94	3 attempts per layer	75.6
4	3 attempts per layer	81
4.06	3 attempts per layer	86.8
4.13	3 attempts per layer	92.9
4.19	3 attempts per layer	99.5
4.25	3 attempts per layer	107
4.31	3 attempts per layer	114
4.38	3 attempts per layer	122
4.44	3 attempts per layer	131
4.5	3 attempts per layer	140
4.56	3 attempts per layer	150
4.63	3 attempts per layer	161
4.69	3 attempts per layer	172
4.75	3 attempts per layer	185
4.81	3 attempts per layer	198
4.88	3 attempts per layer	212
4.94	3 attempts per layer	227
5	3 attempts per layer	243
5.06	3 attempts per layer	260
5.13	3 attempts per layer	279
5.19	3 attempts per layer	299
5.25	3 attempts per layer	320
5.31	3 attempts per layer	343
5.38	3 attempts per layer	367
5.44	3 attempts per layer	393
5.5	3 attempts per layer	421
5.56	3 attempts per layer	451
5.63	3 attempts per layer	483
5.69	3 attempts per layer	517
5.75	3 attempts per layer	554
5.81	3 attempts per layer	593
5.88	3 attempts per layer	635
5.94	3 attempts per layer	681
6	3 attempts per layer	729

Worked examples, recomputed

Every number the mechanisms quote for this relationship is recomputed from it, so none of them disagrees with it.

Where it appears

Each mechanism states it in its own names. The general symbol each name stands for is shown with it.

Data Movement
How do records cross between two systems that have independent rates, independent availability and independent lifetimes, and what are you willing to lose when the assumption underneath the crossing breaks?
$2 \times 2 \times 2 = 8 attempts$
Read it in place →
Distributed Systems Coordination
Who is allowed to decide, and how does everyone else learn the decision without asking?
$1 + r$
1 + r (attempts per request) = $a$
Read it in place →
Fault Tolerance & Delivery Semantics
What is the cheapest thing that can be safely done twice?
$3 \times 3 \times 3 = 27$
Read it in place →
Emergent Failure Dynamics
What happens when nothing is broken, every component is healthy, and the system's own response to stress makes the stress worse?
$(1 + r)^{d}$
1 + r (attempts per tier) = $a$ · d (tiers) = $L$
Read it in place →
Emergent Failure Dynamics
What happens when nothing is broken, every component is healthy, and the system's own response to stress makes the stress worse?
$3 \times 3 \times 3 = 27 s; 4^{3} = 64 s$
Read it in place →