Logging

What is it? #

Logs are the record of what your system did. During an incident they are usually the only evidence you have.

Most logging is bad in one of two ways: too little, so you cannot reconstruct events, or too much undifferentiated noise, so you cannot find anything.

Structured logging fixes most of this. Instead of a sentence, each line is a set of fields, which means you can search and aggregate them.

The other essential is a correlation ID: one identifier attached to every log line for a single request, so you can follow it across services.

Think of it like this #

A flight recorder versus a pilot's memory. When something goes wrong, you do not want "I think it was around four o'clock" — you want timestamped instrument readings.

But a recorder that stored everything at the highest resolution for years would be unreadable. What matters is recording the right things with enough structure to search.

Simple example #

An order fails in production. With good logs you find the request by its correlation ID and see validation passed, payment returned a 502, and the retry also failed. With bad logs you have "Error occurred" and a stack trace with no context.

Code #

import json
import logging
import time
import uuid
from contextvars import ContextVar

request_id: ContextVar[str] = ContextVar("request_id", default="-")


class JsonFormatter(logging.Formatter):
    REDACT = {"password", "token", "authorization", "card_number", "otp"}

    def format(self, record: logging.LogRecord) -> str:
        payload = {
            "ts": time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(record.created)),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
            "request_id": request_id.get(),
        }
        for key, value in getattr(record, "extra_fields", {}).items():
            payload[key] = "[redacted]" if key.lower() in self.REDACT else value
        if record.exc_info:
            payload["error"] = self.formatException(record.exc_info)
        return json.dumps(payload)


handler = logging.StreamHandler()          # stdout: the container collects it
handler.setFormatter(JsonFormatter())
logger = logging.getLogger("app")
logger.addHandler(handler)
logger.setLevel(logging.INFO)


def log(level: int, message: str, **fields) -> None:
    logger.log(level, message, extra={"extra_fields": fields})


# Usage
request_id.set(str(uuid.uuid4()))
log(logging.INFO, "order created", order_id="A-1", user_id=7, total=2499)
log(logging.WARNING, "payment retry", order_id="A-1", attempt=2, gateway="stripe")

{"ts":"2026-09-22T10:15:03","level":"INFO","logger":"app",
 "message":"order created","request_id":"6f1c...","order_id":"A-1",
 "user_id":7,"total":2499}

Levels, used consistently

DEBUG     detail for development; usually off in production
INFO      notable events: order created, job completed, user registered
WARNING   something unexpected but handled: a retry, a fallback, a slow query
ERROR     an operation failed and a user or job was affected
CRITICAL  the service cannot function: database unreachable, out of disk

Never log: passwords, tokens, API keys, card numbers, OTPs, full personal records.

How it works #

The formatter emits one JSON object per line. Log aggregation tools parse that automatically, so order_id=A-1 becomes a searchable field rather than text buried in a sentence.

ContextVar holds the request ID for the current request without passing it through every function. Middleware sets it at the start of each request, and every log line from that request carries it.

That correlation ID is what turns scattered lines into a story. In a system with several services, propagating it in a header lets you trace one user action across all of them.

Redaction is applied in the formatter, which is the right place: a single implementation protects every log call, rather than relying on each developer to remember.

Logging to stdout is deliberate. In containers and under systemd, the platform collects standard output and routes it, so the application does not manage files, rotation or shipping.

Levels only work if used consistently. If everything is INFO, the level is useless for filtering; if errors are logged as warnings, alerts miss them.

Real-world use #

Production logs go to a central system — CloudWatch, Loki, the Elastic stack, Datadog — where they can be searched across all instances. Logs on one server are nearly useless when there are twenty.

Cost becomes real at volume. Verbose debug logging in production can cost more than the servers, which is why sampling is common for high-volume paths.

Retention is a policy decision balancing debugging needs, cost and privacy law. Logs containing personal data fall under data protection rules, which is another reason to keep them minimal and redacted.

Logs are one of three signals. Metrics tell you something is wrong, logs tell you what happened, and traces show where time went. The next lesson covers metrics; together they make incidents tractable.

Common mistakes #

• Logging secrets, tokens or personal data, creating a compliance problem.
• Unstructured messages that cannot be searched or aggregated.
• No correlation ID, so lines from concurrent requests are interleaved and unusable.
• Using the wrong level, so real errors hide among informational noise.
• Writing to local files on ephemeral servers, where the logs vanish with the instance.

Practice #

Add JSON logging with a per-request ID to a small application, including redaction of a sensitive field. Log one INFO, one WARNING and one ERROR line for a single request, then confirm you can filter by request ID and by level.