Dependency cascade

How does availability compose over hard dependencies, and how do you make one soft?

Start with the arithmetic, because it is unforgiving. A service with N hard dependencies (services it cannot work without), each independently available with probability a, has availability a^N.

You cannot build a four-nines (99.99%) service on top of a hundred three-nines services, and no amount of care inside your own code changes that, because your own code is not in the equation.

So the count of hard dependencies is a budget, spent whether or not you track it. Exactly three moves reduce the exponent, and there are no others:

1. Remove the dependency.
2. Make it soft: degrade instead of failing when it is absent.
3. Cache its answer so a failure is survivable for the cache lifetime.

The second is the one that matters in practice. The standard failure is not that teams do not know this. It is that dependencies become hard by accident. The recommendations call is obviously optional, but it sits synchronously in the render path with no fallback. It is now a hard dependency with the same weight in the exponent as the database. Every dependency is hard by default. Softness is a property you must build, and then test, because a fallback path that has never run in production is not a fallback path.

# A soft dependency: bounded wait, a fallback, and a counter so the fallback is visible.
def render_home(rider):
    try:
        recs = recommendations.get(rider, timeout_ms=50)
    except (Timeout, Unavailable):
        metrics.incr("recs.fallback_used")
        recs = DEFAULT_RECS            # an exercised, tested fallback
    return build_page(rider, recs)