Fault Tolerance

What is it? #

Fault tolerance is designing so that failures in one part do not cascade into total failure.

Availability is about redundancy of components. Fault tolerance is about how your code behaves when something it depends on is slow, broken or unreachable.

The dangerous case is not a dependency that fails fast. It is one that hangs. Requests pile up waiting, your workers are exhausted, and a problem in one downstream service takes your whole application down.

Four techniques cover most of it: timeouts, bounded retries with backoff, circuit breakers and fallbacks.

Think of it like this #

A restaurant where the dessert supplier fails to deliver. A fault-tolerant restaurant removes dessert from the menu for the evening and keeps serving.

A fragile one has every waiter standing at the back door waiting for the delivery, so nobody gets their main course either.

Simple example #

Your product page calls a recommendation service. It becomes slow, every request waits 30 seconds, your worker pool fills, and the entire site goes down — because of an optional feature.

Code #

import time
import requests


# 1. Always set a timeout. This is the single most important line.
def fetch_recommendations(user_id: int) -> list:
    response = requests.get(
        f"https://recs.internal/users/{user_id}",
        timeout=(1.0, 2.0),          # 1s to connect, 2s to read
    )
    response.raise_for_status()
    return response.json()


# 2. Circuit breaker: stop calling a service that is clearly broken
class CircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_seconds=30):
        self.failure_threshold = failure_threshold
        self.recovery_seconds = recovery_seconds
        self.failures = 0
        self.opened_at = None

    def call(self, func, *args, **kwargs):
        if self.opened_at is not None:
            if time.time() - self.opened_at < self.recovery_seconds:
                raise CircuitOpen("circuit is open; not calling")
            self.opened_at = None            # half-open: allow one trial call

        try:
            result = func(*args, **kwargs)
        except Exception:
            self.failures += 1
            if self.failures >= self.failure_threshold:
                self.opened_at = time.time()
            raise
        else:
            self.failures = 0                # success closes the circuit
            return result


class CircuitOpen(Exception):
    pass


# 3. Fallback: degrade instead of failing
breaker = CircuitBreaker()

def product_page(product_id: int, user_id: int) -> dict:
    product = database.fetch_product(product_id)     # essential
    try:
        recommendations = breaker.call(fetch_recommendations, user_id)
    except Exception:
        recommendations = popular_products()          # optional: degrade
    return {"product": product, "recommendations": recommendations}

The four techniques

timeout          never wait indefinitely; every network call needs one
retry + backoff  a bounded number of attempts, with increasing delay and jitter
circuit breaker  stop calling a failing service; let it recover
bulkhead         separate resource pools, so one dependency cannot exhaust everything

Plus one decision per dependency: essential, or degrade?

How it works #

The timeout is the foundation. Without it, a hanging dependency holds your worker forever. The connect and read timeouts are separate because they fail for different reasons.

The circuit breaker counts failures. After the threshold, it opens and calls fail immediately without touching the network. That protects two things: your own workers, which stop waiting, and the struggling service, which stops receiving traffic while it recovers.

After the recovery period, the breaker allows one trial call. Success closes it; failure opens it again. That is the half-open state, and it is how the system recovers automatically.

The fallback is where the design decision lives. The product itself is essential — no fallback is possible, so that failure must surface. Recommendations are optional, so a generic list is better than an error page.

Retries deserve care. They must be bounded, spaced with exponential backoff, and jittered so all clients do not retry in unison. Retrying immediately against an overloaded service makes the incident worse, and retrying a non-idempotent operation can duplicate it.

Bulkheads complete the set: separate connection pools or worker groups per dependency, so a slow service can only exhaust its own share.

Real-world use #

Cascading failure is how small incidents become outages. One slow service fills the callers' thread pools, those callers become slow, and their callers fill up in turn.

Service meshes and libraries such as Resilience4j and Polly provide these patterns as configuration. Cloud SDKs usually have built-in retry and timeout settings, which are worth reviewing rather than accepting by default.

Load shedding is the related idea: when overloaded, reject some requests quickly rather than accepting everything and serving nothing. A fast 503 is better than a timeout.

Testing this requires deliberately breaking things — chaos engineering in its practical form. Turning off a dependency in staging reveals whether your fallbacks actually work, which is usually not what people expect.

The most valuable habit is simply asking, for each dependency: what happens if this is slow, and what happens if it is gone?

Common mistakes #

• No timeout on a network call, which is the root cause of most cascading failures.
• Unbounded retries, which turn a small outage into a self-inflicted denial of service.
• Retrying without backoff and jitter, so every client retries in unison.
• Treating optional dependencies as essential, so any failure breaks the page.
• Never testing failure paths, so fallbacks are broken when they are finally needed.

Practice #

Take an endpoint that calls an external service. Add an explicit timeout, a circuit breaker with a threshold and recovery period, and a fallback. Then simulate the service being unreachable and confirm the endpoint still responds quickly with degraded content.