Message Queues

What is it? #

A message queue holds tasks between the code that creates them and the code that performs them.

Instead of doing slow work during a request, you put a message on a queue and return immediately. A separate worker picks it up and does the job.

Three benefits follow. Requests stay fast. Traffic spikes are absorbed, because the queue grows instead of the system falling over. And work survives restarts, because the queue is durable.

The cost is that the work is no longer immediate, and you must design for messages arriving twice or out of order.

Think of it like this #

A restaurant order rail. The waiter clips the order and returns to the floor rather than standing in the kitchen waiting.

If twenty orders arrive at once, the rail holds them. Nothing is forgotten, the waiter is never blocked, and the kitchen works through them at its own pace.

Simple example #

Signing up triggers a welcome email, a CRM sync and thumbnail generation. Doing all three during the request makes signup slow and fragile. Queued, signup takes 80 milliseconds and the rest happens behind the scenes.

Code #

                    ┌──────────┐        ┌──────────┐
  request ─────────▶│ producer │───────▶│  QUEUE   │
  (returns fast)    │ (web app)│        └────┬─────┘
                    └──────────┘             │
                                    ┌────────┼────────┐
                                    ▼        ▼        ▼
                                worker-1 worker-2 worker-3
                                    │
                            failures after N tries
                                    ▼
                             dead letter queue

# Producer: enqueue and return
import json, redis

queue = redis.Redis(decode_responses=True)


def enqueue(task: str, payload: dict) -> None:
    queue.rpush("jobs", json.dumps({
        "task": task,
        "payload": payload,
        "id": payload.get("idempotency_key"),
        "attempts": 0,
    }))


def signup(email: str) -> dict:
    user = create_user(email)                       # the essential work
    enqueue("send_welcome_email", {"user_id": user.id, "idempotency_key": f"welcome:{user.id}"})
    enqueue("sync_to_crm", {"user_id": user.id})
    return {"id": user.id}                          # returns immediately


# Worker: take one message, do the work, handle failure
HANDLERS = {"send_welcome_email": send_welcome_email, "sync_to_crm": sync_to_crm}
MAX_ATTEMPTS = 3


def run_worker():
    while True:
        _, raw = queue.blpop("jobs")                 # blocks until a job arrives
        job = json.loads(raw)
        try:
            if job["id"] and queue.set(f"done:{job['id']}", "1", nx=True, ex=86400) is None:
                continue                             # already processed: skip
            HANDLERS[job["task"]](**job["payload"])
        except Exception as exc:
            job["attempts"] += 1
            if job["attempts"] < MAX_ATTEMPTS:
                queue.rpush("jobs", json.dumps(job))         # retry at the back
            else:
                queue.rpush("jobs:dead", json.dumps(job))    # give up, keep it
            print(f"job failed ({job['attempts']}): {exc}")

Delivery guarantees

at-most-once    may be lost, never duplicated      rarely what you want
at-least-once   never lost, may be duplicated      the common default
exactly-once    expensive and often an illusion

Because at-least-once is the norm, handlers must be idempotent:
processing the same message twice must have the same result as once.

How it works #

enqueue writes a small JSON message and returns. The request finishes without waiting for the email or the CRM call.

blpop blocks until a job is available, so workers do not busy-poll. Several workers can run in parallel, and the queue hands each message to one of them.

The idempotency check uses set(..., nx=True), which succeeds only if the key does not exist. If the same job ID is processed twice — after a crash, or a duplicate delivery — the second attempt is skipped.

That matters because almost all queues deliver at least once. A worker that crashes after doing the work but before acknowledging will see the message again.

Failed jobs are re-queued at the back, not the front, so one poisonous message cannot block everything behind it. After a few attempts they move to a dead letter queue.

The dead letter queue is not optional in practice. It is where you look when something silently is not happening, and it preserves the message so it can be fixed and replayed.

Real systems add exponential backoff between retries, so a temporarily unavailable downstream service is not hammered.

Real-world use #

Emails, notifications, image and video processing, report generation, data exports, webhook delivery, search indexing and payment reconciliation are all queued work in typical systems.

RabbitMQ, Amazon SQS, Redis-based queues and Kafka all fill this role with different trade-offs. Kafka in particular is a durable log rather than a simple queue, which suits event streaming and replay.

Queue depth is the metric that matters most. Growing depth means workers cannot keep up, and it is the earliest warning before user-visible delays.

The failure mode to design for is the queue outage itself. If enqueueing fails, does the user request fail too? For a welcome email, probably not — log it and move on. For a payment webhook, it must not be dropped, so the message goes into the database in the same transaction and is published afterwards.

Common mistakes #

• Handlers that are not idempotent, so a duplicate delivery charges twice or sends two emails.
• No dead letter queue, so failing jobs retry forever or vanish.
• Re-queuing failures at the front, letting one bad message block the queue.
• No monitoring on queue depth, so a backlog goes unnoticed.
• Putting work in a queue that the user is waiting for anyway.

Practice #

Build a small queue with Redis: a producer that enqueues jobs, a worker with three retries and a dead letter queue, and an idempotency check. Make one handler fail deliberately and confirm the job ends up in the dead letter queue after three attempts rather than blocking the others.