Autoscaling oscillation
Why does an autoscaler oscillate, and why is it not an overload defence?
An autoscaler is a controller that adds or removes servers based on a metric. It is a negative-feedback controller (it pushes against change) with dead time d, the total delay between the real state and the effect of the decision:
- the metric window,
- plus the decision interval,
- plus provisioning latency,
- plus image pull,
- plus JIT warm-up (the runtime compiling hot code),
- plus cache warm-up.
Typically this is minutes. The controller acts on a state that is minutes old. A negative-feedback loop with dead time d and enough gain oscillates with a period of roughly 2d. Flapping is not a tuning failure. It is the default behaviour of a delayed controller with enough gain.
Adding capacity first removes capacity
There is a second, sharper problem. Adding capacity during overload first removes capacity. New instances start cold. They miss the cache (the 1/(1−h) leverage of a cache stampede), open a connection storm against the shared database, run interpreted code until the JIT warms, and use upstream capacity to fetch their own state. For the first interval, a scale-out event is a net increase in load. So during an overload the autoscaler is briefly positive feedback. For stateful systems it is worse: scaling changes which node owns which key and triggers rebalancing, and moving state consumes exactly the capacity whose shortage started the scaling.
Two consequences follow. The first is the one most often misbelieved.
Autoscaling is not an overload defence. A controller cannot regulate anything faster than its own delay (autoscaling), and during an overload its first effect is to remove capacity, so the fast time scale belongs to shedding, concurrency limits and backpressure (Overload defences). Nor can it help a hot key: new servers take other shards' work, and the hot key still hashes to one node (skew).
The control law must be asymmetric. Scale up fast and scale down slowly. Being over-provisioned costs money. Being under-provisioned costs an outage. The two are not comparable. This is the same asymmetry as AIMD (additive increase, multiplicative decrease), appearing for the second time, which turns it from a congestion-control trick into a general principle for control when the two kinds of error have different costs. Separate up and down thresholds are required, not optional. This is the hidden-state principle: when a unit's cost depends on recent state, the system has memory and needs two thresholds, and a single threshold guarantees chatter.
Warm pools and predictive scaling both attack d, not the gain. A warm pool (spare, already-started instances) shrinks the provisioning term. Predictive scaling removes the sensing term by scheduling capacity from a known demand shape (for example, the evening peak). Over-provisioning is the honest escape: pay money to leave the control problem altogether.