Logs in Production

What is it? #

Logs are the record of what your system did. During an incident they are frequently the only evidence available.

Two decisions determine whether they help: what you log, and how you structure it. Everything else follows.

Volume is the constraint people underestimate. At scale, logging costs real money, and verbose logging in production can exceed the cost of the servers producing it.

Retention is a policy question involving debugging needs, cost and data protection obligations.

Think of it like this #

A ship's log rather than a recording of every conversation on board.

The log notes what happened, when, and who was involved. A complete recording would be unusable, expensive to store and awkward if anyone asked what personal information you were keeping.

Simple example #

An order fails. With good logs you find the request by its ID, see validation passing, the payment gateway returning a 502 and the retry also failing. With poor logs you have "Error" and a stack trace with no context.

Code #

What to log, at which level

ERROR     an operation failed and a user or job was affected
          include: what failed, the identifiers involved, the cause

WARN      something unexpected but handled — a retry, a fallback,
          a slow query, an approaching limit

INFO      notable state changes: order placed, user registered,
          job completed, deployment started
          NOT: every function entry and exit

DEBUG     detailed diagnostics; off in production, or sampled

Never log: passwords, tokens, API keys, card numbers, OTPs,
full request bodies, or personal data beyond an identifier.

# Structured, with context that makes it searchable
logger.info("order_placed", extra={
    "order_id": order.id,
    "user_id": user.id,
    "total": order.total,
    "payment_method": order.method,
    "request_id": request_id.get(),
})

logger.error("payment_failed", extra={
    "order_id": order.id,
    "gateway": "stripe",
    "status_code": 502,
    "attempt": 2,
    "request_id": request_id.get(),
}, exc_info=True)

# Not this: unsearchable, and it leaks the payload
# logger.info(f"Processing order {order.id} with data {request_body}")

Controlling volume

sample debug logs        log 1% of successful requests in detail,
                         100% of failures
avoid per-item logging   one line per batch, not per row
drop health checks       they are the highest-volume, lowest-value lines
rate limit repeats       a failing loop can produce millions of identical lines
set levels per module    a chatty library at WARN, your code at INFO

Retention, balanced

application logs   14-30 days — enough to investigate most issues
access logs        30-90 days — traffic analysis and abuse investigation
audit logs         1 year or more — who did what, often a requirement
debug logs         1-3 days

Longer retention costs money and increases exposure if logs contain
personal data. Shorter retention makes some investigations impossible.
Decide deliberately rather than by default.

How it works #

Structured logs with named fields can be searched and aggregated. "Show every payment_failed for gateway stripe in the last hour" is a query against fields, not a text search through prose.

Including identifiers — order, user, request — is what makes a log line useful. A message saying an operation failed, without saying which one, tells you only that something is wrong.

The request ID ties every line from one request together, across services. Without it, concurrent requests interleave into noise.

Level discipline is what makes filtering work. If everything is INFO, the level conveys nothing; if errors are logged as warnings, alerts based on errors miss them.

Sampling is the main tool against volume. Full detail on failures and a small sample of successes preserves almost all diagnostic value at a fraction of the cost.

Rate limiting repeated messages matters more than it sounds. A retry loop failing every hundred milliseconds produces hundreds of thousands of identical lines per hour, which can fill a disk or a log budget quickly.

The prohibition on logging secrets and full payloads is not theoretical. Logs are widely readable within a team, retained for months, and frequently shipped to third-party services.

Real-world use #

Centralised logging is necessary as soon as there is more than one server. Logs on one machine are of limited use when three are serving traffic.

Log costs surprise teams regularly. Hosted logging is usually priced by ingested volume, and one verbose deployment can multiply the bill.

Correlation IDs propagated across services are what make distributed debugging tractable. Generating one at the edge and passing it in a header is a small change with large returns.

Logs containing personal data fall under data protection rules, which means retention limits and, potentially, deletion requests. Logging an identifier rather than the data itself avoids most of this.

Logs answer "what happened". Metrics answer "is something wrong now", and error tracking answers "what is broken and how often". All three are covered in this track, and they are complementary rather than alternatives.

Common mistakes #

• Unstructured messages that cannot be searched or aggregated.
• Logging secrets, tokens or full request bodies.
• No request ID, so concurrent requests interleave into unusable noise.
• Verbose logging in production, producing surprising bills.
• No rate limiting on repeated errors, so a failing loop floods the system.

Practice #

Take an existing log output and rewrite three lines as structured events with identifiers and a request ID. Then estimate the daily volume your application produces, identify the highest-volume line, and decide whether to sample, drop or keep it.