Rate Limiting in Production

What is it? #

Rate limiting protects capacity, ensures fairness between users, and controls cost when each request consumes money downstream.

In production it is applied in layers: a CDN or WAF at the edge, the load balancer or gateway, and the application where it has the most context.

The algorithms were covered in the system design track. This lesson is about deployment decisions: which layer, which key, which limit, and what to return.

The most important detail is the response. A 429 with Retry-After lets a well-behaved client back off correctly; an unexplained failure produces an immediate retry and makes things worse.

Think of it like this #

Crowd control at an event. Barriers outside the venue handle the obvious surge, staff at the door manage the flow, and staff inside handle the specific areas that get crowded.

Trying to do all of it at one point produces a queue at that point.

Simple example #

An API allows 1,000 requests per minute per key in general, 10 per minute for report generation because each costs money downstream, and 5 login attempts per 15 minutes per account.

Code #

Layers, and what each is good at

CDN / WAF          volumetric abuse, obvious bots, geographic blocks
                   cheapest place to drop traffic: it never reaches you

load balancer      coarse per-IP limits, connection limits
                   protects the application tier from a flood

API gateway        per-key and per-route limits, quota tiers
                   knows the client identity, not the business context

application        per-user, per-endpoint, per-resource limits
                   the only layer that knows what the request means

Each layer catches what the one before it could not.

# Nginx: a reasonable second layer
limit_req_zone $binary_remote_addr zone=general:10m rate=30r/s;
limit_req_zone $binary_remote_addr zone=login:10m   rate=1r/m;
limit_conn_zone $binary_remote_addr zone=conns:10m;

server {
    limit_conn conns 20;                         # concurrent connections per IP

    location / {
        limit_req zone=general burst=50 nodelay; # allow short bursts
        limit_req_status 429;
        proxy_pass http://app;
    }

    location /login {
        limit_req zone=login burst=3;
        limit_req_status 429;
        proxy_pass http://app;
    }
}

Choosing the key

API key / user ID    fairest for authenticated traffic; per-customer quotas
IP address           the only option for anonymous traffic, but shared
                     connections mean one office can hit the limit together
account + action     login attempts per account, to stop distributed
                     credential stuffing across many IPs
route + user         expensive endpoints get their own tighter limit

Limiting only by IP punishes users behind shared networks and fails
against attackers with many addresses.

The response

HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1758542400

{"error": "rate_limited", "detail": "Try again in 30 seconds"}

Without Retry-After, clients retry immediately and increase the load
they were limited for. Publish limits in your documentation so clients
can pace themselves rather than discovering limits by failing.

Load shedding: the emergency case

When genuinely overloaded, rejecting some requests quickly is better
than accepting everything and serving none of them.

priority   keep health checks and payment completion working
shed       drop expensive, optional endpoints first
fast fail  a quick 503 beats a 30-second timeout

This is a deliberate choice made in advance, not an accident.

How it works #

Layering works because each layer sees something the others cannot. The edge sees volume and geography; the gateway sees the client identity; the application knows whether the request is an expensive report or a cheap lookup.

Dropping traffic at the edge is the cheapest option, because it never consumes your resources at all.

The Nginx burst parameter allows a short burst above the sustained rate, which matters because real clients are bursty. A page load firing eight requests at once should not be limited when the sustained rate is fine. nodelay serves the burst immediately rather than queueing it.

Key choice is a fairness decision with real consequences. IP-based limits are the only option for anonymous traffic and penalise shared connections; per-account limits are what actually stop credential stuffing spread across many addresses.

Per-endpoint limits reflect cost. An endpoint that triggers a paid third-party call deserves a much tighter limit than a cached read.

The response headers let good clients self-regulate, which reduces 429s for everyone. Publishing the limits in documentation does the same.

Load shedding is the deliberate version of failing under load. Deciding in advance which endpoints are sacrificial means an overload degrades the optional parts rather than everything at once.

Real-world use #

Every public API has limits, and the ones with clear documentation and headers generate far fewer support requests.

Login endpoints deserve their own strict limit regardless of the general policy, keyed by account as well as by IP, because credential stuffing uses many addresses against one account.

Cost control is an underrated reason. An endpoint calling a paid API per request can produce a surprising bill if one client loops on it, and a per-user limit prevents that.

Legitimate clients do get caught, particularly behind corporate networks and mobile carriers where many users share an address. Per-user limits for authenticated traffic avoid most of it.

Monitoring the 429 rate is worth doing. A rise can mean abuse, or it can mean a legitimate integration that outgrew its limit and needs a conversation rather than a block.

Common mistakes #

• Returning 429 with no Retry-After, so clients retry immediately.
• Limiting only by IP, punishing shared networks and failing against distributed attacks.
• The same limit for cheap and very expensive endpoints.
• No strict limit on login and password reset endpoints.
• Per-instance counters, so the effective limit multiplies with the server count.

Practice #

Define a rate limiting plan for an API: which layer enforces what, which key each limit uses, and the limits for a general endpoint, an expensive endpoint and login. Then implement one layer and verify the 429 response includes accurate Retry-After and limit headers.