High Availability

What is it? #

High availability means the system keeps serving users when individual components fail.

The core technique is removing single points of failure. Anything that exists once — one server, one database, one load balancer, one region — is something whose failure takes everything down.

Availability is usually expressed in nines. 99.9% allows about 43 minutes of downtime a month; 99.99% allows about four. Each extra nine costs substantially more.

The honest question is not "how do we get five nines" but "what does an hour of downtime actually cost us, and what are we willing to spend to avoid it".

Think of it like this #

A hospital generator. The mains supply is reliable, but not reliable enough for an operating theatre, so there is a second source that takes over automatically.

The generator costs money and is tested monthly even though it is rarely needed. That is what availability engineering looks like: paying for redundancy you hope stays idle.

Simple example #

A system with one application server, one database and one load balancer has three single points of failure. Any one of them failing takes the whole service down.

Code #

Single points of failure, and the usual answers

one app server        → several behind a load balancer
one load balancer     → a redundant pair, or a managed service
one database          → primary with a replica and automatic failover
one availability zone → spread across at least two
one region            → multi-region, expensive and complex
one deploy pipeline   → the ability to deploy manually when it breaks
one person who knows  → documentation and runbooks

What the nines mean

99%      ~7.2 hours downtime per month      one server, best effort
99.9%    ~43 minutes per month              redundancy plus monitoring
99.99%   ~4.3 minutes per month             multi-zone, automatic failover
99.999%  ~26 seconds per month              multi-region, huge investment

Note: planned maintenance counts. So does a slow deploy.

# Failover for a dependency: try the healthy option, degrade rather than fail
class ResilientCache:
    def __init__(self, primary, fallback=None):
        self.primary = primary
        self.fallback = fallback

    def get(self, key):
        try:
            return self.primary.get(key)
        except ConnectionError:
            metrics.increment("cache.primary_unavailable")
            if self.fallback is not None:
                return self.fallback.get(key)
            return None            # a cache miss, not an outage


def render_product(product_id):
    cached = cache.get(f"product:{product_id}")
    if cached:
        return cached
    # If the cache is down, the page is slower but still works
    return database.fetch_product(product_id)

Design questions worth asking about each component

- if this fails, what breaks and for whom?
- how quickly do we notice?
- does failover happen automatically, or does a human do it?
- have we ever tested the failover?
- what is the degraded experience: an error, or something slower but usable?

How it works #

Redundancy is necessary but not sufficient. A second database that nobody can promote automatically, and whose failover has never been tested, is documentation rather than availability.

Detection speed dominates real-world downtime. If failover takes 30 seconds but detection takes 20 minutes, your availability is determined by the monitoring, not the redundancy.

The cache example shows graceful degradation, which is often more valuable than more redundancy. A failed cache should make the site slower, not broken. Deciding in advance which dependencies are essential and which are optional is a design activity.

Availability zones are the practical middle ground: separate physical facilities within one region, with low latency between them. Spreading across two zones survives a facility failure without the complexity of multi-region.

Multi-region is genuinely hard, because data must be replicated across long distances, and you must decide what happens when the regions cannot see each other. Most systems do not need it.

Planned maintenance counts as downtime from the user's perspective, which is why zero-downtime deployment techniques matter as much as hardware redundancy.

Real-world use #

Most outages are not hardware failures. They are bad deployments, configuration mistakes, expired certificates, full disks and exhausted connection pools. Redundancy does not protect against a bad configuration rolled out everywhere.

That is why deployment safety — staged rollouts, health checks, fast rollback — contributes more to real availability than an extra replica. The CI/CD track covers those directly.

Failover must be tested. Organisations that practise it deliberately, including by injecting failures, are the ones whose failover works when it is needed.

The cost conversation should be explicit. If an hour of downtime costs a small amount and happens twice a year, spending heavily on multi-region infrastructure is the wrong trade. For a payments platform the calculation is completely different.

Dependencies deserve the same scrutiny. Your availability cannot exceed that of the third-party services you require to be up.

Common mistakes #

• Redundancy that has never been tested with a real failover.
• Fast failover behind slow detection, so the outage lasts anyway.
• Treating every dependency as essential, so any failure becomes a full outage.
• Ignoring planned maintenance and deploys when measuring availability.
• Buying more nines than the business actually needs.

Practice #

Draw your system and mark every component that exists only once. For each, write what happens when it fails, how quickly you would notice, and whether recovery is automatic. Then pick the cheapest change that removes the most risk.