Horizontal Scaling

What is it? #

Horizontal scaling means handling more load by adding more machines rather than by making one machine bigger.

It has two advantages over a bigger machine: there is no ceiling, and several machines give you redundancy as a side effect.

The requirement is that your application servers must be stateless and interchangeable. Any instance must be able to handle any request, which means nothing important lives in local memory or on the local disk.

The bottleneck then moves. Application servers scale easily; the database, which holds the shared state, does not.

Think of it like this #

Opening more checkout tills rather than training one cashier to work faster.

For that to work, every till needs access to the same prices and stock information. If one till kept its own private price list, customers would get different answers depending on where they queued.

Simple example #

An application runs on one server at 80% CPU during peak hours. Instead of a bigger machine, three identical instances run behind a load balancer, and an autoscaler adds a fourth when traffic rises.

Code #

What must move out of the application server

sessions        → Redis or a signed token
uploaded files  → object storage
cached data     → shared cache, not process memory
scheduled jobs  → one designated runner or a distributed lock
in-memory state → a database or shared store
local logs      → stdout, collected centrally

# BAD: state that lives in one process
SESSIONS = {}               # invisible to other instances
UPLOAD_DIR = "./uploads"    # only on this machine
CACHE = {}                  # each instance has a different view

@app.post("/login")
def login(credentials):
    sid = new_id()
    SESSIONS[sid] = credentials.user_id    # the next request may hit another server
    return {"session": sid}


# GOOD: shared state, disposable servers
import redis
sessions = redis.Redis(host="redis.internal")

@app.post("/login")
def login(credentials):
    sid = new_id()
    sessions.setex(f"session:{sid}", 1209600, credentials.user_id)
    return {"session": sid}

# Autoscaling on a signal that reflects real load
minReplicas: 2                 # never one: two gives redundancy
maxReplicas: 20
metrics:
  - type: Resource
    resource:
      name: cpu
      target: { type: Utilization, averageUtilization: 70 }
  - type: Pods                 # often better than CPU for queue-driven work
    pods:
      metric: { name: jobs_pending_per_worker }
      target: { type: AverageValue, averageValue: "30" }

Where the bottleneck goes next

10 app servers  → 10x the database connections
                → 10x the cache traffic
                → 10x the calls to any third-party API
                → scheduled jobs running 10 times unless controlled

Scaling the stateless tier is easy. Everything shared behind it is the real work.

How it works #

The bad example stores sessions in a dictionary in one process. With two servers, a user logs in on server A and the next request goes to server B, which has never heard of them. The symptom is intermittent logouts, which is maddening to debug.

The good version stores the session in Redis, so every instance sees the same data. The server itself holds nothing that cannot be recreated.

Once servers are stateless, they become disposable. You can destroy one mid-request and the load balancer retries elsewhere. That property is what makes rolling deployments and autoscaling safe.

Autoscaling on CPU works for CPU-bound web traffic. For queue workers, scaling on pending jobs per worker matches the real signal much better — CPU may be low while a backlog builds.

A minimum of two instances is deliberate. One instance means a restart is an outage, whatever the scaling configuration says.

The final block is the point people discover late. Ten application servers means ten times the connections to your database. Connection poolers, caches and per-service limits are what stop the shared tier collapsing under a successful scale-out.

Real-world use #

Cloud platforms and container orchestrators make adding instances trivial. Making an application ready for it is where the work is, and it is mostly about removing local state.

The twelve-factor guidelines describe this well: treat processes as stateless and disposable, keep configuration in the environment, and write logs to stdout. Following them makes scaling a configuration change.

Scheduled jobs are the classic scaling bug. Cron on every instance means a nightly report sent five times, which is visible to customers and embarrassing.

Scaling down is as important as scaling up. Instances must handle SIGTERM by finishing in-flight work, or every scale-down event drops requests.

Cost also drives this. Several small instances that scale with demand are usually cheaper than one large machine sized for peak traffic and idle most of the day.

Common mistakes #

• Keeping sessions, uploads or caches in process memory or on local disk.
• Running scheduled jobs on every instance.
• Connection pools that multiply across instances and exhaust the database.
• A minimum of one replica, so restarts cause downtime.
• No graceful shutdown, so scale-down and deployments drop requests.

Practice #

Audit a service you know and list everything it keeps locally: sessions, files, caches, timers, in-memory counters. For each, write where it should live instead. Then describe what a user would experience if you scaled to three instances without moving them.