Background Workers

What is it? #

A worker is a process that does work outside the request cycle. It either pulls jobs from a queue or runs on a schedule.

Workers are separate from web servers because the two have different needs. Web servers must respond in milliseconds; workers may run for minutes. Scaling them independently is the whole point.

The practical concerns are concurrency, timeouts, graceful shutdown, visibility and scheduling. Each of these is a common source of production problems.

A worker that vanishes quietly, or runs the same scheduled job on five servers at once, causes problems that are hard to notice and harder to diagnose.

Think of it like this #

A back office behind a shop counter. The counter serves customers quickly; the back office handles paperwork, deliveries and end-of-day reconciliation.

If the counter had to do all of that, queues would form. If the back office silently stopped working, nobody at the counter would notice for days.

Simple example #

An application has web servers handling requests, a pool of workers consuming a job queue, and a scheduler running nightly reports and cleanup. Each is deployed and scaled separately.

Code #

Three kinds of background work

queue workers      consume jobs as they arrive        — scale with queue depth
scheduled jobs     run at fixed times                 — must run exactly once
long-running       stream consumers, watchers         — must restart on failure

import signal
import time

running = True


def handle_shutdown(signum, frame):
    """Finish the current job, then exit. Do not abandon work mid-flight."""
    global running
    print("shutdown requested; finishing current job")
    running = False


signal.signal(signal.SIGTERM, handle_shutdown)      # sent by systemd / Kubernetes
signal.signal(signal.SIGINT, handle_shutdown)


def worker_loop(queue, handlers, job_timeout=300):
    while running:
        job = queue.reserve(timeout=5)              # returns None if idle
        if job is None:
            continue

        started = time.monotonic()
        try:
            handlers[job.task](**job.payload)
            queue.ack(job)
        except Exception as exc:
            queue.retry_or_dead_letter(job, exc)
        finally:
            duration = time.monotonic() - started
            metrics.observe("job_duration_seconds", duration, task=job.task)
            if duration > job_timeout:
                logger.warning("slow job", extra={"task": job.task, "seconds": duration})

    print("worker exited cleanly")

# systemd unit: keep the worker running and restart it if it dies
[Unit]
Description=App background worker
After=network.target

[Service]
User=appuser
WorkingDirectory=/srv/app
Environment=PYTHONUNBUFFERED=1
ExecStart=/srv/app/.venv/bin/python -m app.worker
Restart=always
RestartSec=5
KillSignal=SIGTERM
TimeoutStopSec=60          # allow the current job to finish
[Install]
WantedBy=multi-user.target

Scheduled jobs across several servers

WRONG   cron on every server  → the nightly report runs five times

RIGHT   one designated scheduler, or a lock:
        if cache.set("lock:nightly-report", "1", nx=True, ex=3600):
            run_nightly_report()

How it works #

The signal handler is the part most often missing. When a deployment or an autoscaler stops a worker, it sends SIGTERM. Without a handler, the process dies immediately and any in-flight job is abandoned halfway.

Setting running = False lets the loop finish the current job and exit cleanly. TimeoutStopSec=60 in the unit file gives it time to do so before being killed forcibly.

queue.reserve(timeout=5) blocks briefly rather than spinning, so an idle worker uses no CPU while still checking the shutdown flag regularly.

Acknowledging only after success is what makes at-least-once delivery work: a crash mid-job leaves the message unacknowledged, so another worker will pick it up.

Recording job duration turns invisible work into something you can see. Without metrics, a job that has silently grown from two seconds to two minutes goes unnoticed until the queue backs up.

The scheduling note addresses a genuinely common bug. Cron configured on every application server means every job runs once per server. A single designated scheduler, or a distributed lock, fixes it.

Real-world use #

Celery, RQ, Sidekiq and cloud-managed workers are the usual tools. Container platforms run workers as separate deployments scaled by queue depth rather than by request rate.

Scheduled work covers nightly reports, subscription renewals, cleanup of expired data, backup verification and reconciliation jobs. Each needs the run-exactly-once guarantee.

Workers need their own monitoring. Response time and error rate mean nothing here; queue depth, job duration, failure rate, dead letter count and worker liveness are the metrics that matter.

Deployment is where graceful shutdown proves itself. Rolling out a new version stops every worker in turn, and without clean shutdown, each restart abandons whatever it was doing — which for payment or email jobs means duplicates or losses.

Common mistakes #

• No SIGTERM handling, so deployments abandon in-flight jobs.
• Running cron on every server, so scheduled jobs execute several times.
• No metrics on job duration and failures, leaving problems invisible.
• Workers sharing a process with the web server, so slow jobs block requests.
• Unbounded job runtime, letting one stuck job occupy a worker indefinitely.

Practice #

Write a worker loop with SIGTERM handling that finishes the current job before exiting. Add duration logging and a slow-job warning. Then write a systemd unit that restarts it automatically and allows 60 seconds for shutdown.