Circuit breakers and half-open probing

What does a breaker really protect, and where does its hysteresis live?

A circuit breaker is a client-side classifier of the callee's state. It has three positions:

• Closed: requests pass through.
• Open: requests fail immediately, without calling the callee.
• Half-open: a small number of probe requests pass to test whether the callee has recovered.

The trip condition is an error rate over a time window, not a raw count: raw counts trip on high traffic and never trip on low traffic. That is the most common bug. A minimal sketch of the three states:

import time

class CircuitBreaker:
    def __init__(self, error_rate_limit=0.5, min_calls=20, cooldown=10.0):
        self.state = "closed"; self.opened_at = 0.0
        self.calls = 0; self.errors = 0
        self.error_rate_limit, self.min_calls, self.cooldown = error_rate_limit, min_calls, cooldown

    def call(self, fn, fallback):
        if self.state == "open":
            if time.time() - self.opened_at < self.cooldown:
                return fallback()                  # fail fast, no attempt
            self.state = "half-open"               # allow a probe
        try:
            result = fn()
        except Exception:
            self._record(error=True); return fallback()
        self._record(error=False); return result

    def _record(self, error):
        self.calls += 1; self.errors += int(error)
        if self.state == "half-open":
            self.state = "open" if error else "closed"
            self.opened_at = time.time(); self.calls = self.errors = 0
        elif self.calls >= self.min_calls and self.errors / self.calls >= self.error_rate_limit:
            self.state = "open"; self.opened_at = time.time(); self.calls = self.errors = 0

(This sketch ignores the time window and the jitter on the half-open step. It only shows the three states.)

Its real function is subtler than "stop calling a dead service." Open mode turns a slow failure into a fast one and, this is the point, returns the caller's resources. Recall the exhaustion cascade: the damage that a slow dependency does is proportional to how long callers hold threads and connections while waiting for it. A breaker caps that at the detection threshold. It protects the caller first and the callee second.

Half-open is where the hysteresis lives. It is the hidden-state principle made concrete: a system with hidden state needs separate entry and exit thresholds, and the exit threshold must be lower, because a recovering callee is at C_cold, not C_warm. A breaker that closes straight back to full traffic triggers the overload again at once. That is the restart problem in miniature. The right behaviour is one probe, then a ramp: progressive readmission, the same shape as slow-start and staggered warm-up.

Breakers are synchronized deciders. All clients see the same degradation and open together. So a partly degraded service gets zero traffic and cannot show that it has recovered, and then gets all traffic at once when the clients close together. Add jitter to the thresholds and to the probe timing. The thundering-herd fix applies to control decisions, not only to requests.

Getting it wrong

• All-or-nothing. A hard "open" is a cliff: it cannot say "this dependency is at half capacity". Pair the breaker with adaptive concurrency limits, which degrade continuously, or use gradual or adaptive breaking, which is strictly better: reject a percentage of requests in proportion to the observed failure rate. Netflix's client-side adaptive throttling does this: it rejects with a probability derived from the ratio of requests to successes. It recovers smoothly instead of swinging between open and closed.
• Wrong granularity. One breaker for a whole service trips when one endpoint or one shard is bad, and it kills the healthy paths. Breakers per endpoint, per shard or per instance are usually right.
• No fallback. The open state must produce something: a cached value, a default, a degraded response, or a clean error. "Open" with no fallback is only a faster failure. That is sometimes correct, but decide it on purpose.
• Breaking on the wrong signal. Trip on your health indicators: timeouts, connection failures, 5xx errors. Do not trip on 4xx errors. A client that sends bad requests does not mean the dependency is broken.

Timeouts are the precondition. A circuit breaker cannot detect a hang without a timeout. A timeout without a breaker means every request pays the full timeout. They are a pair. Both must be set from the dependency's real latency distribution (roughly its p99.9), not from round numbers.