What is it? #
Monitoring is measuring your system continuously so you notice problems early and can see what changed.
Metrics are numbers over time: request rate, error rate, latency, queue depth, CPU. They are cheap to store and fast to query, which makes them the right tool for "is something wrong".
Logs explain what happened; metrics tell you that it is happening at all. You need both.
Alerts turn metrics into action. The hard part is alerting on things that genuinely need a human, at a threshold that catches real problems without waking people for noise.
Think of it like this #
The dashboard in a car. A few instruments always visible — speed, fuel, temperature — and a warning light for the things that need action now.
A car with sixty gauges would be ignored. A car with no oil pressure warning would be destroyed. The design of what is shown, and what triggers a light, is the whole skill.
Simple example #
Checkout starts failing for 5% of users. Without monitoring you find out from support tickets hours later. With it, an error-rate alert fires in two minutes and a dashboard shows the payment service latency climbing just before.
Code #
The four signals worth measuring on every service
latency how long requests take — measured in percentiles
traffic requests per second
errors failed requests as a share of the total
saturation how close to a limit: CPU, memory, connections, queue depth# Prometheus-style instrumentation
from prometheus_client import Counter, Histogram, Gauge
import time
REQUESTS = Counter("http_requests_total", "Requests", ["method", "path", "status"])
LATENCY = Histogram("http_request_seconds", "Request duration", ["path"],
buckets=(0.05, 0.1, 0.25, 0.5, 1, 2, 5))
QUEUE_DEPTH = Gauge("jobs_pending", "Jobs waiting in the queue")
def timed_request(method: str, path: str, handler):
started = time.perf_counter()
status = 500
try:
response = handler()
status = response.status_code
return response
finally:
LATENCY.labels(path=path).observe(time.perf_counter() - started)
REQUESTS.labels(method=method, path=path, status=status).inc()Why percentiles, not averages
10 requests: nine take 100ms, one takes 5000ms
average = 590ms (describes nobody's experience)
p50 = 100ms (the typical user)
p95 = 5000ms (the user who is suffering)
p99 = 5000ms
Averages hide the slow tail. Alert on p95 or p99.# An alert worth having: symptom-based, with a duration
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.02
for: 5m # must persist, to avoid flapping
labels: { severity: page }
annotations:
summary: "Over 2% of requests are failing"
runbook: "https://wiki.internal/runbooks/high-error-rate"How it works #
A counter only increases and answers "how many". A histogram records a distribution, which is what makes percentiles possible. A gauge goes up and down and answers "how much right now".
Labels turn one metric into many views: errors by endpoint, latency by path. They are powerful and must be used carefully — a label with unbounded values, such as a user ID, creates a separate time series per user and will overwhelm your metrics system.
The finally block records the metric whatever happens, including on an exception, so failures are counted rather than silently missing.
The percentile explanation is the single most useful idea in this lesson. Averages are dominated by the bulk of fast requests and hide the users having a bad time. p95 and p99 are where real complaints live.
The alert rule fires on a symptom users feel — a failing share of requests — rather than on a cause such as high CPU. High CPU with healthy responses does not need a human at 3am.
The for: 5m clause requires the condition to persist, which prevents alerts flapping on brief spikes. The runbook link is what makes the alert actionable for whoever receives it.
Real-world use #
Prometheus with Grafana is the common open-source combination; Datadog, New Relic and cloud-native monitoring offer the same ideas as a service.
The cultural side matters more than the tooling. Alert fatigue is real: when most pages are noise, people stop reading them, and the one that mattered is missed. Every alert should be something a human must act on now.
Service level objectives formalise this. Deciding that 99.9% of requests should succeed within 500ms gives you a concrete number to alert on and an error budget to spend on releases.
Tracing is the third signal alongside metrics and logs. It shows where the time went inside one request across services, which is how you find that 300ms of a 400ms response was one slow downstream call.
Dashboards are for investigation, alerts are for detection. Keep the alert list short and the dashboards rich.
Common mistakes #
- Alerting on averages, which hide the slow tail users experience.
- Using unbounded label values such as user IDs, exploding the number of time series.
- Alerting on causes like CPU rather than on symptoms users feel.
- So many alerts that people ignore them.
- No runbook, so whoever is paged has to work out what to do from scratch.
Practice #
Instrument an endpoint with a request counter and a latency histogram. Produce p50, p95 and p99 from real traffic and note the difference. Then write one alert rule that fires only when the failure rate exceeds 2% for five minutes, and draft a three-line runbook for it.