Load Balancer

What is it? #

A load balancer sits in front of several servers and spreads incoming requests across them.

It solves two problems at once. Capacity: no single machine has to handle everything. Availability: if one server dies, traffic goes to the others and users notice nothing.

It decides where to send each request using an algorithm — round robin, least connections, or weighted variants — and it checks the health of each server continuously.

For this to work, your servers must be interchangeable. Any server must be able to handle any request, which means no session data in local memory and no files written to local disk.

Think of it like this #

A supermarket queue manager directing customers to whichever till is free. One queue, many tills.

If a till closes, the manager simply stops sending people there. Customers do not need to know which till exists, and no till holds a customer's half-finished basket.

Simple example #

Traffic grows past what one server can handle. Instead of a bigger machine, you run three identical servers and put a load balancer in front. DNS points at the load balancer, not at any individual server.

Code #

                    ┌──────────────┐
   users ──────────▶│ load balancer│  (the public address)
                    └──────┬───────┘
              ┌────────────┼────────────┐
              ▼            ▼            ▼
         ┌─────────┐  ┌─────────┐  ┌─────────┐
         │ app-1   │  │ app-2   │  │ app-3   │   identical servers
         └────┬────┘  └────┬────┘  └────┬────┘
              └────────────┼────────────┘
                           ▼
                   shared database / cache / storage

# Nginx as a load balancer
upstream app_servers {
    least_conn;                                   # send to the least busy
    server 10.0.0.11:3000 max_fails=3 fail_timeout=15s;
    server 10.0.0.12:3000 max_fails=3 fail_timeout=15s;
    server 10.0.0.13:3000 max_fails=3 fail_timeout=15s backup;
}

server {
    listen 443 ssl;
    server_name example.com;

    location / {
        proxy_pass http://app_servers;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 3s;
        proxy_next_upstream error timeout http_502;   # retry elsewhere
    }
}

# Every app server needs a health endpoint the balancer can poll
@app.get("/health")
def health():
    # Check what this instance genuinely needs to serve traffic
    db_ok = check_database()
    return ({"status": "ok"}, 200) if db_ok else ({"status": "degraded"}, 503)

Algorithms

round robin        each server in turn — simple, fine when requests are similar
least connections  the server with fewest active requests — better for uneven work
IP hash            same client always to the same server — a form of stickiness
weighted           bigger servers receive proportionally more

How it works #

The load balancer owns the public address. DNS points at it, and clients never learn the addresses of individual servers.

least_conn sends each request to whichever server currently has fewest active connections. That handles uneven request durations better than plain round robin.

max_fails and fail_timeout implement passive health checking: after three failures within the window, that server is taken out of rotation for a while. Dedicated load balancers also poll a health endpoint actively.

The health endpoint should check what the instance actually needs. Returning 200 unconditionally means a server with a broken database connection keeps receiving traffic.

The X-Forwarded-For and X-Forwarded-Proto headers pass along the original client IP and protocol. Without them, your application sees every request as coming from the load balancer over plain HTTP, which breaks logging, rate limiting and redirect logic.

proxy_next_upstream retries a failed request on another server, which turns a single server crash into a slightly slower request instead of an error.

The backup server receives traffic only when the others are unavailable — useful for a smaller standby instance.

Real-world use #

Every system beyond a single server has one, whether it is Nginx, HAProxy, a cloud load balancer, or the routing layer of a container platform.

The requirement that servers be interchangeable shapes application design. Sessions move to Redis or signed tokens, uploaded files go to object storage, and scheduled jobs run in one designated place rather than on every instance.

Sticky sessions — pinning a user to one server — are available and usually a mistake. They defeat even load distribution and cause visible failures when that server restarts. They are a workaround for state that should not be local.

Load balancers are also where TLS usually terminates, where rate limiting often lives, and where you can shift traffic gradually during a deployment — the mechanism behind blue-green and canary releases in the CI/CD track.

Common mistakes #

• Storing sessions or uploads on local disk, so behaviour depends on which server answers.
• A health endpoint that always returns 200, keeping broken instances in rotation.
• Forgetting X-Forwarded-For, so every request appears to come from the load balancer.
• Using sticky sessions to paper over local state.
• Running one load balancer with no redundancy, recreating a single point of failure.

Practice #

Run two copies of a small application on different ports and put Nginx in front with least_conn. Add a health endpoint you can switch to failing, then confirm the balancer stops sending traffic there and resumes when it recovers.