Alerting is where monitoring meets a human being, usually at the worst possible time. A team that pages on every CPU spike learns to ignore its pager, and then misses the one page that mattered. A team that pages only when users are affected, with enough context to act, sleeps better and responds faster. This module shows you how to alert on symptoms instead of causes, how to build the multi-window burn-rate alerts that SLO-based teams rely on, and how to cut an inherited pile of noisy alerts down to the few that deserve to wake someone.
- Distinguish symptom-based from cause-based alerts, and decide what deserves a page, a ticket or nothing
- Explain why simple threshold alerts on error rate perform badly, in terms of precision, recall, detection time and reset time
- Build multi-window, multi-burn-rate alerts from an SLO, and verify the arithmetic behind the thresholds
- Write Prometheus alerting rules and route them with Alertmanager grouping, inhibition and silences
- Run a regular alert review that reduces noise and keeps every page actionable
Symptoms, not causes
A symptom is something users experience: requests failing, pages loading slowly, data going stale. A cause is something inside the system that may or may not lead to a symptom: high CPU, a restarted pod, a full connection pool, a failed node.
Cause-based alerts fail in both directions. They fire when nothing is wrong for users: a pod restarts and Kubernetes replaces it, or CPU sits at 90% on a batch host that is meant to be busy. And they miss real problems, because you can never list every possible cause in advance. The failure that hurts you is, almost by definition, the one you did not think to write an alert for. A symptom-based alert on the user's experience catches it anyway, whatever lies behind it.
| Cause-based (avoid as a page) | Symptom-based (page on this) |
|---|---|
| CPU above 90% for 5 minutes | Checkout error budget is burning 14 times too fast |
| A pod restarted | Fewer than half the expected requests are being served |
| Database connections above 80% | 99th percentile latency has breached the SLO threshold for 10% of requests |
| Disk at 85% | The nightly export has not completed by its deadline |
Causes still matter, in two places. They belong on dashboards, where they help the person who was paged find out why. And a few causes deserve tickets, not pages, when they predict a future symptom with time to spare: a disk that will fill in four days, or a certificate that expires in two weeks. Nobody needs to be woken for those, but somebody must deal with them during working hours.
Every notification should fall into one of three classes, and the test for the first is strict.
| Class | Test | Delivery |
|---|---|---|
| Page | Users are affected now, or will be very soon, and a human must act immediately | Pager, phone call; wakes people up |
| Ticket | A human must act, but it can wait until working hours | Ticket queue, reviewed daily |
| Log / dashboard | No action is needed; it is context for investigation | Nowhere. Not email, not a chat channel nobody reads |
An alert that goes to an email folder or a muted chat channel is not an alert. It is noise that trains everyone to ignore notifications, and it gives a false sense of coverage. If nobody needs to act, delete it.
Why simple thresholds disappoint
The Google SRE Workbook evaluates alerting strategies on four properties.
- Precision: the share of alerts that corresponded to a real, significant event. Low precision means false alarms.
- Recall: the share of significant events that produced an alert. Low recall means missed incidents.
- Detection time: how long a problem lasts before the alert fires.
- Reset time: how long the alert keeps firing after the problem is fixed.
Take an SLO of 99.9%, so the allowed error rate is 0.1%. The obvious alert is "error rate above 0.1% over the last 10 minutes". It has good recall and fast detection, but terrible precision. A ten-minute blip at 0.15% errors fires the pager, yet it consumes only about 0.035% of a thirty-day budget. At that sensitivity you could be paged over a hundred times a month and still meet the SLO comfortably.
Lengthen the window to 36 hours and precision improves, but now a total outage takes far too long to detect, and the alert keeps firing for a day and a half after the fix. Adding a for: 1h clause fails differently: the condition has to hold continuously, so an outage that flaps resets the timer each time and never alerts, and a complete outage still goes unreported for a full hour.
The root of the problem is that error rate alone is the wrong quantity. What you care about is how much error budget is being consumed, and how fast. A high error rate for two minutes and a slightly raised rate for three days can do the same damage to the SLO, and they call for different urgency. That is what burn rate captures.
Burn-rate alerts
From the SLO module: burn rate is how fast you consume the budget relative to the rate that would spend it exactly over the SLO period. At burn rate 1 the budget lasts the whole period. At burn rate 10 it is gone in a tenth of it.
burn rate = observed error rate / (1 - SLO)
budget consumed = burn rate x alert window / SLO period
burn rate threshold = budget share you are willing to lose x SLO period / alert window
For a 30-day SLO period (720 hours):
lose 2% of the budget in 1 hour -> 0.02 x 720 / 1 = 14.4
lose 5% of the budget in 6 hours -> 0.05 x 720 / 6 = 6
lose 10% of the budget in 3 days -> 0.10 x 720 / 72 = 1
With SLO 99.9% (allowed error rate 0.1%), the error-rate thresholds are:
14.4 x 0.1% = 1.44% 6 x 0.1% = 0.6% 1 x 0.1% = 0.1%That gives the recommended starting set: two pages for fast burns and one ticket for a slow burn. The fast-burn alerts catch outages, and the slow one catches the quiet, persistent degradation that would otherwise consume the budget without anyone noticing.
One problem remains. An alert over a one-hour window keeps firing for up to an hour after the errors stop, which is a poor reset time, and it invites people to distrust the alert. The fix is a second, short window, conventionally one twelfth of the long one, evaluated against the same threshold. The alert fires only when both exceed it.
| Severity | Windows (long + short) | Burn rate | Budget consumed |
|---|---|---|---|
| Page | 1 hour + 5 minutes | 14.4 | 2% |
| Page | 6 hours + 30 minutes | 6 | 5% |
| Ticket | 3 days + 6 hours | 1 | 10% |
The long window establishes that the burn is significant, and the short window confirms that it is still happening. Once the problem is fixed, the five-minute rate drops within minutes and the alert clears, even though the one-hour average is still high. This is the multi-window, multi-burn-rate alert, and it scores well on all four properties at once.
These numbers are a starting point for a 30-day period, not a law. If your period is 28 days, recompute the thresholds with 672 hours. If a service has very little traffic, a handful of failed requests can produce a huge burn rate; low-traffic services need longer windows, synthetic traffic to raise the volume, or a minimum request count in the alert condition.
Implementing it in Prometheus
First record the error ratio over each window you need, so that the alert expressions stay short and the dashboard uses the same definition.
groups:
- name: checkout-slo-recording
rules:
- record: job:slo_errors_per_request:ratio_rate5m
expr: |
sum by (job) (rate(http_requests_total{job="checkout", code=~"5.."}[5m]))
/
sum by (job) (rate(http_requests_total{job="checkout"}[5m]))
- record: job:slo_errors_per_request:ratio_rate1h
expr: |
sum by (job) (rate(http_requests_total{job="checkout", code=~"5.."}[1h]))
/
sum by (job) (rate(http_requests_total{job="checkout"}[1h]))
# ...repeat for 30m, 6h and 3dgroups:
- name: checkout-slo-alerts
rules:
- alert: CheckoutErrorBudgetFastBurn
expr: |
(
job:slo_errors_per_request:ratio_rate1h{job="checkout"} > (14.4 * 0.001)
and
job:slo_errors_per_request:ratio_rate5m{job="checkout"} > (14.4 * 0.001)
)
or
(
job:slo_errors_per_request:ratio_rate6h{job="checkout"} > (6 * 0.001)
and
job:slo_errors_per_request:ratio_rate30m{job="checkout"} > (6 * 0.001)
)
labels:
severity: page
team: checkout
annotations:
summary: "Checkout is burning its error budget fast"
description: "Error ratio over 1h is {{ $value | humanizePercentage }}. SLO is 99.9%."
runbook_url: "https://runbooks.example.com/checkout/error-budget-burn"
dashboard: "https://grafana.example.com/d/checkout"
- alert: CheckoutErrorBudgetSlowBurn
expr: |
job:slo_errors_per_request:ratio_rate3d{job="checkout"} > (1 * 0.001)
and
job:slo_errors_per_request:ratio_rate6h{job="checkout"} > (1 * 0.001)
labels:
severity: ticket
team: checkout
annotations:
summary: "Checkout is slowly exhausting its error budget"0.001is1 - SLOfor 99.9%. Keeping it visible in the expression documents where the threshold came from.- Labels drive routing:
severityandteamdecide who is notified and how. Annotations carry information for the human: what is happening, how bad it is, and links. - Every page needs a runbook link and a dashboard link. The person paged at three in the morning should be one click from what to do and one click from the picture.
- The
for:clause is unnecessary here, because the long window already provides the smoothing. Where you do usefor:, remember that the condition must hold continuously for that whole time. - Unit-test the rules with
promtool test rules, feeding in a synthetic series and asserting which alerts fire. Open-source generators such as Sloth and Pyrra produce the whole rule set from a short SLO definition.
Routing, grouping and silencing
Prometheus decides whether an alert is firing. Alertmanager decides who hears about it, and how. It deduplicates, groups related alerts into one notification, routes by label, suppresses alerts that are consequences of others, and honours silences.
route:
receiver: default-tickets
group_by: [alertname, team]
group_wait: 30s # wait to collect related alerts before the first notification
group_interval: 5m # wait before notifying about new alerts added to the group
repeat_interval: 4h # re-notify if still firing
routes:
- matchers: ['severity="page"']
receiver: oncall-pager
- matchers: ['severity="ticket"']
receiver: default-tickets
inhibit_rules:
- source_matchers: ['alertname="ClusterUnreachable"']
target_matchers: ['severity="page"']
equal: [cluster]
receivers:
- name: oncall-pager
pagerduty_configs:
- routing_key_file: /etc/alertmanager/secrets/pagerduty-key
- name: default-tickets
webhook_configs:
- url: http://ticket-bridge:8080/alerts- Grouping turns fifty alerts from one failing cluster into a single notification listing fifty instances, instead of fifty pages.
- Inhibition mutes alerts that are downstream consequences. If the whole cluster is unreachable, there is no value in also being told about every service inside it.
- Silences mute matching alerts for a fixed period, for planned maintenance. Always give them an expiry, an author and a comment, and never use one to hide a noisy alert indefinitely. Fix or delete the alert instead.
- Escalation, acknowledgement and schedules live in the paging tool. If the primary does not acknowledge within a few minutes, the page goes to the secondary.
amtool check-config alertmanager.yml
amtool alert query severity=page
amtool silence add alertname=CheckoutErrorBudgetFastBurn \
--duration 2h --author jane --comment "planned database failover, change CHG-1042"
amtool silence query
promtool check rules rules/checkout-slo-alerts.ymlWho watches the watcher? Add one alert that always fires, often called a dead man's switch, and route it to an external heartbeat service. If the heartbeats stop arriving, that service pages you, which covers the case where Prometheus, Alertmanager or the network path between them has failed silently.
Killing alert fatigue
Alert fatigue sets in when people are paged so often, or for so little, that they stop responding with care. It is a safety problem, not merely an annoyance: fatigued responders acknowledge and ignore, and the one real page is lost among the false ones. The Google SRE book suggests that an on-call shift should see no more than about two incidents, which leaves time to handle each one properly and follow up afterwards.
Hold a short alert review every week or two. Go through every page since the last review and ask the same questions of each.
- Was it actionable? Did the responder do something that mattered, beyond acknowledging it?
- Was it urgent? Could it have waited until morning as a ticket?
- Did it reflect user impact, or an internal condition that users never noticed?
- Was it a duplicate of another alert for the same underlying problem?
- Did the alert carry enough context, with a runbook that helped?
Then act on the answers. Delete alerts that were never actionable. Demote the non-urgent to tickets. Replace groups of cause-based alerts with one SLO burn-rate alert. Fix the underlying problem for alerts that fire repeatedly with the same manual remedy, because that is toil. Rewrite alerts that lacked context. Track pages per shift, the share outside working hours, and the share that was actionable, and expect all three to improve.
Deleting an alert feels risky, which is why noisy alerts survive. The symptom-based SLO alerts are your safety net: if they are in place and trusted, a removed cause-based alert cannot hide a user-facing problem. It can only remove a warning that you were not acting on anyway.