Docker in Production

What is it? #

Running containers in production adds concerns that development does not surface: signal handling, resource limits, log collection, health checks and updating without dropping requests.

The subtlest is PID 1. The first process in a container is treated specially by the kernel, and if it does not handle signals properly, your container cannot shut down gracefully.

Resource limits are not optional. Without them, one container can exhaust the host's memory and take everything else down with it.

Updating without downtime on a single host means starting the new container, waiting for it to become healthy, and only then retiring the old one.

Think of it like this #

The difference between a workshop prototype and equipment on a factory floor.

The prototype works. Production adds guards, limits, emergency stops and a procedure for changing a part without halting the line.

Simple example #

An application container in production: it handles SIGTERM, has memory and CPU limits, reports health, logs to stdout with rotation, and is updated by starting a replacement before stopping the original.

Code #

# PID 1 and signals — the subtle one
# BAD: shell form wraps the process in /bin/sh, which does not forward SIGTERM
# CMD python -m app.main

# GOOD: exec form — the application is PID 1 and receives signals directly
CMD ["python", "-m", "app.main"]

# If the entrypoint must be a script, exec into the real process
# entrypoint.sh:
#   #!/bin/sh
#   set -e
#   python manage.py migrate
#   exec python -m app.main        # exec REPLACES the shell, keeping PID 1

# The application must handle SIGTERM
import signal, sys

def shutdown(signum, frame):
    server.stop(grace=25)      # finish in-flight requests
    sys.exit(0)

signal.signal(signal.SIGTERM, shutdown)

# Production Compose settings
services:
  app:
    image: registry.example.com/team/myapp:sha-a1b2c3d   # immutable tag
    restart: unless-stopped
    stop_grace_period: 30s            # matches the app's shutdown time

    deploy:
      resources:
        limits:   { cpus: '2.0', memory: 1G }
        reservations: { memory: 256M }

    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:8000/health"]
      interval: 15s
      timeout: 3s
      start_period: 30s
      retries: 3

    logging:
      driver: json-file
      options: { max-size: "50m", max-file: "5" }     # or logs fill the disk

    read_only: true                                    # immutable filesystem
    tmpfs: [/tmp]
    security_opt: [no-new-privileges:true]
    cap_drop: [ALL]

# Updating without downtime on a single host
docker run -d --name app-new --network shop-net registry.example.com/team/myapp:sha-new

for i in {1..30}; do
    docker exec app-new curl -fsS http://localhost:8000/health && break
    [ "$i" = 30 ] && { docker rm -f app-new; echo "new version unhealthy"; exit 1; }
    sleep 2
done

# switch the proxy upstream to app-new, then retire the old one
docker stop -t 30 app-old && docker rm app-old
docker rename app-new app

Production checklist for containers

[ ] exec-form CMD; the app handles SIGTERM
[ ] stop grace period longer than the app's shutdown time
[ ] memory and CPU limits set
[ ] health check that verifies real dependencies
[ ] log rotation configured on the logging driver
[ ] immutable tag or digest, never latest
[ ] non-root user, read-only filesystem, capabilities dropped
[ ] data in named volumes, and those volumes backed up
[ ] image scanned, base image current

How it works #

PID 1 in Linux does not get default signal handlers. If the process there ignores SIGTERM, the container is killed after the grace period instead of shutting down cleanly, dropping in-flight requests on every deployment.

Shell-form CMD runs /bin/sh -c, which becomes PID 1 and does not forward signals to the child. Exec form avoids that, and an entrypoint script must use exec for the same reason.

stop_grace_period is the container equivalent of TimeoutStopSec. It must exceed the application's own shutdown time, or the graceful handling is cut short anyway.

Memory limits turn a leak into a container restart rather than a host-wide OOM event affecting other services. Reservations help the scheduler place the container sensibly.

Health checks must verify real dependencies. A container whose process is alive but cannot reach the database should report unhealthy so it is replaced or removed from routing.

Log rotation on the driver is essential. The default json-file driver has no size limit, and container logs filling the disk is a common incident.

The hardening options — read-only filesystem, dropped capabilities, no new privileges — are cheap and meaningfully reduce what a compromised container can do.

The update sequence starts the replacement, waits for health, switches traffic and only then stops the old container. Stopping first would mean downtime for as long as the new container takes to start.

Real-world use #

Most production container deployments use an orchestrator, which provides rolling updates, health-based routing and restart policies as built-in features. The concerns above are the same; the orchestrator implements them.

On a single host, Compose plus a reverse proxy covers a surprising amount, and the manual update sequence above is what the proxy switch automates.

Signal handling is the problem that bites teams unexpectedly. It works fine until traffic is high enough that dropped requests during deployment become visible.

Log volume is genuinely underestimated. A busy application without rotation can fill a disk in days, and the failure arrives as a full host rather than an obvious logging problem.

The container-level hardening options are free and rarely applied. Turning them on in an existing deployment usually takes a short round of fixing paths that needed writing, and is worth the effort.

Common mistakes #

• Shell-form CMD, so the application never receives SIGTERM.
• No resource limits, letting one container exhaust host memory.
• No log rotation, filling the disk with container logs.
• Deploying a moving tag, so nobody knows what is running.
• Stopping the old container before the new one is healthy.

Practice #

Take a container you run and add: exec-form CMD, a SIGTERM handler, memory and CPU limits, a real health check, log rotation and the hardening options. Then perform a zero-downtime update by starting a replacement, waiting for health and retiring the original while sending continuous requests.