Skip to content
About
Main content

Graceful degradation and brownout

When is a smaller answer the right response to overload or to a failed dependency?

is not a yes or a no. Treating it as binary is what makes outages total. If a service has independent features, a failed or overloaded dependency should remove only the features that need it, provided the dependency was declared optional in advance.

Degrade before you refuse

The key idea that separates this from shedding: availability and completeness are separate axes, and users value them very differently. Shedding refuses some users entirely. serves all users a reduced response: recommendations dropped, personalization skipped, a stale cache served, search results unranked, an image at lower resolution. At Wayhop: skip the "ride again?" call (the optional recommendations call from Dependency cascade) and show the booking screen without it.

Its limit: brownout helps only if the optional work is a large share of the cost. Dropping a 2 ms enrichment call does not save a service whose cost is a 200 ms scan. Measure before you design the ladder.

A static classification, not a runtime decision

Each dependency is classified as critical or optional. Each optional one has a defined fallback (a stale cache, an empty result, a default value, a disabled feature). The fallback must be reachable without the failed dependency. This is design-time work, the soft dependency of the arithmetic, decided months earlier. It cannot be added during an incident. Systems without it fail totally: a single unclassified dependency inside a request path fails the whole request, and the classification exists only in someone's head.

Common degradation modes

ModeWhat it dropsWhat it keeps
Read-onlywrite availabilityeverything else; the right default when the write quorum is lost
Stale readfreshnessboth reads and writes
Reduced scopecompleteness: serve only the partitions that are upthe answers it gives
Approximateavailability (exactness is spent to buy it)

Each mode is a declared invariant sacrifice, and the order in which to make them is the most useful summary of this page:

Give up freshness (serve stale data) before completeness, completeness before availability, and availability before correctness. Nearly every defence here is a step down this list. Serving stale data is the first resort because it is the cheapest step.

The rule that makes degradation honest

The response must say it is degraded. A stale or partial answer that looks like a complete one moves the failure into downstream systems, which compute on it silently. In an analytics stack this is worse than an error: an error stops a pipeline, while a silently partial result becomes a number in a report. The correct default depends on the system: Degradation is only correct when the consumer can tell.

The cost nobody budgets

The fallback path runs rarely, so it is the least tested code in the system, and it runs at the most stressful moments. A fallback path that has never run in production is not a fallback path. This is the clearest case of the hypothesis "the recovery path should be the normal path". The right practice is to exercise degraded modes continuously at a low rate, which is the same argument as restarting to test the cold start, and the direct reason chaos testing exists as a practice, not as a preference.