Rate Limiting

What is it? #

Rate limiting caps how many requests a client may make in a period.

It exists for three reasons. Protection, so one caller cannot exhaust your capacity. Fairness, so one heavy user does not degrade everyone else. Cost control, where each request costs money downstream.

The main algorithms are fixed window, sliding window and token bucket. They differ in how they handle bursts and boundaries.

The response matters as much as the limit. Return 429 with a Retry-After header so well-behaved clients back off correctly.

Think of it like this #

A theme park ride with a "three rides per hour" wristband scan. Without it, one enthusiastic group monopolises the ride and everyone else waits.

The scanner also has to be quick and consistent, or the queue for the scanner becomes the new problem.

Simple example #

A public API allows 100 requests per minute per API key. Login attempts are limited far more tightly — five per fifteen minutes per account — because the goal there is stopping credential stuffing rather than managing load.

Code #

import time
import redis

r = redis.Redis(decode_responses=True)


# --- Fixed window: simplest, but allows a burst across the boundary ---
def fixed_window(key: str, limit: int, window: int) -> bool:
    bucket = f"rl:{key}:{int(time.time() // window)}"
    count = r.incr(bucket)
    if count == 1:
        r.expire(bucket, window)
    return count <= limit
# Weakness: 100 requests at 11:59:59 and 100 more at 12:00:01 = 200 in two seconds


# --- Sliding window: counts the actual last N seconds ---
def sliding_window(key: str, limit: int, window: int) -> bool:
    now = time.time()
    bucket = f"rl:sw:{key}"
    pipe = r.pipeline()
    pipe.zremrangebyscore(bucket, 0, now - window)   # drop old entries
    pipe.zadd(bucket, {f"{now}:{time.monotonic_ns()}": now})
    pipe.zcard(bucket)
    pipe.expire(bucket, window)
    _, _, count, _ = pipe.execute()
    return count <= limit


# --- Token bucket: allows controlled bursts, refills steadily ---
class TokenBucket:
    def __init__(self, capacity: int, refill_per_second: float):
        self.capacity = capacity
        self.refill = refill_per_second

    def allow(self, key: str, cost: int = 1) -> tuple[bool, float]:
        now = time.time()
        state = r.hgetall(f"rl:tb:{key}") or {}
        tokens = float(state.get("tokens", self.capacity))
        last = float(state.get("last", now))

        tokens = min(self.capacity, tokens + (now - last) * self.refill)
        allowed = tokens >= cost
        if allowed:
            tokens -= cost

        r.hset(f"rl:tb:{key}", mapping={"tokens": tokens, "last": now})
        r.expire(f"rl:tb:{key}", 3600)
        retry_after = 0.0 if allowed else (cost - tokens) / self.refill
        return allowed, retry_after

The response when a client is limited

HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1758542400

{"error": "rate_limited", "detail": "Try again in 30 seconds"}

What to limit by

API key / user ID     the fairest for authenticated traffic
IP address            for anonymous traffic — but shared IPs affect many users
endpoint + user       expensive endpoints get tighter limits
account + action      login attempts per account, to stop credential stuffing

How it works #

Fixed window is a counter per time bucket. It is cheap and easy, and its weakness is the boundary: a client can use the full quota at the end of one window and again at the start of the next, producing double the intended rate briefly.

Sliding window keeps timestamps in a sorted set, removes entries older than the window and counts what remains. It is accurate, at the cost of storing one entry per request.

Token bucket models a bucket that refills at a steady rate. Each request takes a token; an empty bucket means refusal. It naturally allows a burst up to the capacity while holding the long-run average to the refill rate, which matches how real clients behave.

Computing retry_after from the token deficit gives the client an honest answer rather than a guess.

The counters must be shared across servers, which is why they live in Redis rather than in process memory. Per-instance limits multiply by the number of instances.

Choosing the key is a design decision with real consequences. Limiting by IP address punishes users behind shared connections; limiting by API key is fair but only works for authenticated traffic.

Real-world use #

Every public API enforces limits, usually with the X-RateLimit-* headers so clients can self-regulate rather than discovering the limit by failing.

Login endpoints need tight limits regardless of general traffic policy, and limiting by account rather than only by IP is what stops distributed credential stuffing.

Layering is normal: a CDN or WAF absorbs obvious abuse, the load balancer enforces coarse limits, and the application applies per-user and per-endpoint rules where it knows the context.

Rate limiting also protects downstream services. If your application calls a paid API with a quota, limiting inbound traffic prevents one caller exhausting it for everyone.

A quieter benefit is cost control. Expensive endpoints — report generation, exports, anything that calls a third party per request — deserve their own tighter limits.

Common mistakes #

• Counting in process memory, so the real limit multiplies by the number of servers.
• Returning 429 with no Retry-After, leaving clients to guess and retry immediately.
• Limiting only by IP, which punishes users behind shared connections.
• Using the same limit for cheap and very expensive endpoints.
• Forgetting a strict limit on login and password reset endpoints.

Practice #

Implement a token bucket allowing 10 requests with a refill of 1 per second, backed by Redis. Return 429 with an accurate Retry-After when the bucket is empty. Then add a separate, stricter limit for a login endpoint keyed by account.