What is it? #
Monitoring is measuring the system continuously. Alerting is deciding when a human needs to act on what was measured.
The common failure is measuring plenty and alerting badly: dozens of alerts, most of them noise, and the important one missed among them.
The principle that fixes it is alerting on symptoms rather than causes. High CPU with healthy responses needs no one at 3am; a failing checkout does.
Service level objectives turn this into numbers. Deciding that 99.9% of requests should succeed within 500 milliseconds gives you a concrete threshold and an error budget to spend.
Think of it like this #
A hospital monitor that alarms for a dangerous heart rate, not for every small movement.
An alarm that sounds constantly gets muted, and then the one that mattered is inaudible.
Simple example #
Checkout starts failing for 5% of users. An error-rate alert fires within two minutes, the dashboard shows payment gateway latency rising just before, and the runbook link in the alert says what to check first.
Code #
Measure these on every service
latency p50, p95, p99 — never the average
traffic requests per second
errors failed requests as a proportion of the total
saturation how close to a limit: CPU, memory, connections, queue depth
Plus, for the business: signups, orders, payments. Technical health can
look fine while a broken form quietly stops all conversions.# Alert on symptoms, with duration and a runbook
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.02
for: 5m
labels: { severity: page }
annotations:
summary: "Over 2% of requests failing for 5 minutes"
runbook: "https://wiki.internal/runbooks/high-error-rate"
- alert: CheckoutConversionDropped
expr: |
sum(rate(orders_completed_total[15m]))
< 0.5 * sum(rate(orders_completed_total[15m] offset 1w))
for: 15m
labels: { severity: page }
annotations:
summary: "Orders at less than half of the same time last week"
# Not an alert: CPU above 80%. That is a dashboard line.Service level objectives
SLI what you measure: the proportion of requests succeeding under 500ms
SLO the target: 99.9% over 30 days
error budget 0.1% of requests may fail — about 43 minutes per month
Budget remaining → ship features.
Budget exhausted → stop shipping and improve reliability.
It converts "how reliable should we be" from an argument into a number.Alert severity, honestly
page wake someone now: users are affected and it will not self-resolve
ticket needs attention today: disk filling, certificate expiring soon
log only record it; nobody is notified
If an alert is never acted on, it should be a ticket or deleted.
An alert set that people trust is short.How it works #
Percentiles rather than averages is the most consequential measurement decision. An average is dominated by the fast majority and hides the users having a bad experience; p95 and p99 are where complaints originate.
Alerting on symptoms means alerting on what users experience. Causes belong on dashboards, because there are many possible causes and only a few symptoms worth waking someone for.
The for: duration prevents flapping. A brief spike that resolves before anyone can look should not page anyone.
Business metric alerts catch what technical monitoring misses entirely. A deployment that breaks a form leaves error rates normal and conversions at zero.
Comparing against the same period a week earlier handles daily and weekly traffic patterns, which absolute thresholds cannot.
The error budget reframes reliability as a trade rather than an absolute. If the budget is intact, the system is reliable enough and the team can ship; if it is exhausted, reliability work takes priority. That removes a recurring argument.
Every paging alert needs a runbook. Being woken with a problem and no guidance is how incidents last longer than they should.
Real-world use #
Alert fatigue is the dominant failure mode. Teams that page on everything end up ignoring pages, and the fix is deleting alerts rather than adding more.
A practical review is to look at every alert from the last month and ask whether anyone acted on it. Those that were always ignored should be tickets or removed.
Business metrics deserve equal status with technical ones. Signups, orders and payments are the measurements that reveal whether the system is doing its job.
Tracing is the third signal alongside metrics and logs. It shows where time went inside a single request across services, which is how you find that 300ms of a 400ms response was one slow downstream call.
On-call works when alerts are trustworthy, runbooks exist and the rotation is fair. The technical setup is the easy half.
Common mistakes #
- Alerting on averages, which hide the slow tail users experience.
- Paging on causes such as CPU rather than on user-visible symptoms.
- No duration threshold, so brief spikes page people.
- Monitoring only technical metrics and missing a broken conversion funnel.
- Alerts with no runbook, leaving whoever is paged to start from scratch.
Practice #
List every alert that currently pages your team and mark which were acted on in the last month. Delete or downgrade the rest. Then add one business metric alert comparing against the same period last week, and write a three-line runbook for it.