Module 08: Alerting Without Fatigue

Why this matters

Observability (modules 04-07) lets you investigate a problem once you know to look. Alerting is the other half: being told to look, at the right moment, for the right reasons — without being told so often, or so pointlessly, that you stop listening. That last clause is the whole module. The failure mode of alerting isn't "too few alerts"; it's alert fatigue — so many alerts, so many of them noise, that the on-call engineer mutes the channel, and the one alert that mattered scrolls by unread at 3am next to forty that didn't. Alert fatigue is not a minor annoyance; it is the way good monitoring produces bad outcomes, and it's how real outages get missed by teams who "had an alert for that."

So this module is less about the mechanics of firing an alert (that part is a few lines of PromQL and a routing config) and more about the philosophy: what deserves to wake a human, what doesn't, and how to tell the difference reliably. The central shift is from cause-based alerting ("CPU is at 90%," "a disk is filling," "this service restarted") — which produces mountains of alerts that may or may not matter to anyone — to symptom-based, SLO-driven alerting ("users are experiencing errors," "requests are too slow"), which alerts on the thing you actually care about: whether the service is meeting its promise to users. The metrics you built in module 06 (RED, the golden signals) are exactly the raw material for this.

You'll learn to define SLIs and SLOs, derive an error budget, and alert on burn rate rather than raw thresholds — the modern SRE approach that fires urgently only when the budget is being consumed fast enough to matter, and stays quiet otherwise. You'll learn the hard rule that every page must be actionable (a human, woken up, can and must do something), the difference between things that page and things that can wait for a ticket or a dashboard, and the practices — runbooks, severity levels, grouping, silencing — that keep the signal-to-noise ratio high enough that people still trust the alerts. This is where all the observability plumbing turns into operational readiness, and it's the discipline that decides whether your on-call rotation is sustainable or burns people out.

Concepts

The real enemy: alert fatigue

Start with the problem, because every practice in this module is a response to it. Alert fatigue is the desensitization that happens when people receive too many alerts, especially non-actionable or false ones. Its progression is predictable and lethal:

1. Alerts are added generously — every metric that might indicate a problem gets a threshold ("alert if CPU > 80%," "alert if any 500 occurs").
2. Most of them fire routinely without corresponding to a real, actionable problem (CPU spikes are normal; a single 500 is often harmless).
3. On-call learns that alerts are usually noise, so they start ignoring, muting, or auto-acknowledging them.
4. A real, serious alert fires — and is treated exactly like the noise: ignored. The outage the alert was supposed to catch happens anyway.

The costs compound: missed real incidents, burned-out engineers who dread being on-call, and an erosion of trust that makes the whole monitoring investment worthless. The core realization: an alert's value is not in existing — it's in being trusted and acted on. A noisy alert is worse than no alert, because it actively trains people to ignore the channel where the real alert will appear. Every decision in this module optimizes for one thing: keep the signal-to-noise ratio high enough that every alert is trusted.

Symptom-based vs cause-based alerting

This is the single most important conceptual shift. There are two philosophies for what to alert on:

• Cause-based (avoid as pages): alert on internal conditions that might cause a problem — high CPU, low disk, high memory, a restarted pod, a full queue, a slow query. The trouble: most of these are either self-correcting, normal under load, or don't actually affect users — so they generate huge volumes of alerts, many meaningless. High CPU with happy users and fast responses is not a problem. Alerting on it pages someone for nothing.
• Symptom-based (prefer as pages): alert on what the user actually experiences — requests are failing, requests are too slow, the service is unavailable. These are, by definition, things that matter, because they're defined in terms of user pain. "The error rate users see exceeds 1%" is worth waking someone for; "a CPU is at 85%" usually isn't.

The guiding principle (from Google's SRE practice): page on symptoms, not causes. You want a small number of symptom alerts that mean "users are being hurt," not a large number of cause alerts that mean "something you might care about changed." Causes still have value — but as diagnostic signals on dashboards you consult after a symptom alert fires (module 07's investigation), not as things that page a human. The USE-method resource metrics (module 06) are mostly causes: great for the dashboard you open during an incident, wrong as pages. The RED/golden-signal metrics are mostly symptoms: exactly what should page.

The litmus test for every proposed alert: "If this fires, is a user being hurt (or about to be), and is there something a human must do about it?" If no to either, it shouldn't page.

SLIs, SLOs, and error budgets

Symptom-based alerting needs a precise definition of "the service is doing its job." That's what the SLI/SLO framework provides — the vocabulary of modern reliability.

• SLI (Service Level Indicator): a measurement of some aspect of service health, expressed as a ratio of good events to total. Examples: availability = successful requests / total requests; latency = requests faster than 300ms / total requests. SLIs come straight from your RED metrics (module 06).
• SLO (Service Level Objective): a target for an SLI over a window — the promise. "99.9% of requests succeed over 30 days." "99% of requests complete under 300ms over 30 days." The SLO is a deliberate choice: not 100% (that's impossible and infinitely expensive), but a realistic, agreed level of reliability that keeps users happy while leaving room to ship.
• SLA (Service Level Agreement): the contractual version of an SLO with consequences (refunds, penalties) if breached. SLAs are usually looser than your internal SLOs — you alert on the tighter SLO so you react before the contractual SLA is at risk. (SLA is a business/legal artifact; SLO is what you operate to.)
• Error budget: the mathematical complement of the SLO, and the key insight. If your SLO is 99.9% success, then 100% − 99.9% = 0.1% of requests are allowed to fail over the window. That 0.1% is your error budget — a concrete, spendable quantity of unreliability. Over 30 days at a given traffic level it's a specific number of failed requests you can "afford."

   SLI  ── measure ──▶  successful requests / total          (from RED metrics)
                              │
   SLO  ── target  ──▶  "99.9% succeed over 30 days"          (the promise)
                              │
   ERROR BUDGET = 100% − 99.9% = 0.1%  ── allowed to fail
                              │
   at 1,000,000 req/30d  ──▶  1,000 failures you can "afford"  (spendable)
                              │
   alert when you SPEND IT TOO FAST, not on every single error

The error budget reframes reliability from "never fail" (impossible, paralysing) to "stay within budget" (finite, manageable), and it directly powers good alerting: you don't alert on every error (you have a budget for some), you alert when you're spending the budget too fast to last the window. It also settles the eternal dev-vs-ops tension: budget remaining → ship features; budget exhausted → freeze features and fix reliability. Reliability becomes a number everyone agrees on, not an argument.

Burn rate: alerting on the budget, not raw thresholds

Naive threshold alerting ("page if error rate > 1% for 5 minutes") is both too noisy (a brief blip pages you) and too crude (it treats a tiny sustained leak and a catastrophic spike the same). The modern approach is error-budget burn rate.

Burn rate is how fast you're consuming the error budget relative to the rate that would exactly exhaust it over the window. Burn rate 1 means you'll spend exactly your whole budget by the end of the window (fine — that's the plan). Burn rate 14.4 means you're spending it 14.4× too fast — at that rate the entire 30-day budget is gone in ~2 days. High burn rate = urgent; low burn rate = it can wait.

The power of burn-rate alerting:

• It scales urgency to severity. A massive outage (burn rate 100+) pages immediately; a slow simmer (burn rate 2) creates a ticket for business hours. One SLO, graded responses.
• Multi-window, multi-burn-rate alerts are the SRE-recommended pattern: fire a fast, paging alert only when a high burn rate is sustained over both a short and a longer window (e.g. burn rate ≥ 14.4 over 5m and 1h) — the two windows together kill false positives (a 30-second blip won't satisfy the 1h window) while still catching real fast burns quickly. A lower burn rate (e.g. ≥ 3 over 6h) fires a non-paging ticket. This is precisely the noise-suppression the whole module is about, expressed as math.

   30-sec blip:   5m window ✓ HIGH   1h window ✗ low    ─▶  NO page (false alarm killed)
   fast outage:   5m window ✓ HIGH   1h window ✓ HIGH   ─▶  PAGE now (burn≥14.4, ~2 days to empty)
   slow simmer:   6h window ✓ ≥3     (single window)    ─▶  ticket, business hours
   healthy:       burn ≈ 1  (spending exactly on plan)  ─▶  silence
        short window = FAST to fire   long window = TRUSTWORTHY (no blips)

# error-budget burn rate for a 99.9% availability SLO (budget = 0.001)
# ratio of the observed error rate to the budget:
(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))
) / 0.001
# a paging alert fires when this is >= 14.4 over BOTH 5m and 1h windows

The takeaway: alert on how fast you're burning the promise to users, not on raw metric values crossing arbitrary lines. It's the technical expression of symptom-based alerting plus the error budget.

Actionable alerts, severity, and routing

Beyond what to alert on, how an alert is shaped decides whether it helps or hurts. Rules that keep alerts trustworthy:

• Every page must be actionable. If an alert fires and the right response is "watch it" or "it'll clear up," it should not have paged. A page means: a human, possibly asleep, must do something now. If there's nothing to do, it's not a page — at most a ticket or a dashboard annotation. This single rule eliminates most fatigue.
• Grade severity, route accordingly. Not everything is a 3am page:
  • Page / critical — user-facing, urgent, actionable now → wake someone (PagerDuty/Opsgenie/phone).
  • Ticket / warning — real but not urgent → a ticket or Slack for business hours (e.g. disk will fill in 5 days, a slow budget burn).
  • Info / dashboard — awareness only → a dashboard or a log; never a notification. The severity determines the channel and the urgency, and keeps the paging channel sacred for things that truly need a human now.
• Every alert needs a runbook. A good alert links to a runbook: what it means, how to confirm it's real, and the first diagnostic/remediation steps. An alert with no runbook wakes someone who then doesn't know what to do — the page was half-useless. The runbook is where module 07's investigation loop lives (which trace, which dashboard, which logs).
• Group and deduplicate. One root cause often trips many alerts (a DB down fails ten services). Alerting tools (Alertmanager) group related alerts into one notification and deduplicate repeats, so you get one "DB is down" page, not ten. Grouping is a primary anti-fatigue mechanism.
• Silences and maintenance windows. During a known deploy or maintenance, silence the alerts you expect to fire so they don't cry wolf — deliberately, time-boxed, not by muting the channel permanently.
• Tune continuously. Alerting is never "done." Review fired alerts: which were actionable? which were noise? Delete or re-tune the noisy ones. A regular "alert review" is how you keep the ratio high as the system changes.

The through-line from module 04's log levels to here: don't cry wolf. An ERROR log that isn't really an error, and a page that isn't really actionable, are the same sin — they erode trust in the signal until the real one is ignored.

Command reference

A symptom-based, multi-window burn-rate alert (Prometheus rules):

groups:
- name: slo-availability
  rules:
  # fast burn: page immediately — budget gone in ~2 days at this rate
  - alert: HighErrorBudgetBurnFast
    expr: |
      (sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) / 0.001 >= 14.4
      and
      (sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h]))) / 0.001 >= 14.4
    for: 2m
    labels: {severity: page}
    annotations:
      summary: "Fast error-budget burn on API availability SLO"
      runbook: "https://runbooks.internal/api-availability"
  # slow burn: ticket, not a page — real but not urgent
  - alert: ErrorBudgetBurnSlow
    expr: |
      (sum(rate(http_requests_total{status=~"5.."}[6h])) / sum(rate(http_requests_total[6h]))) / 0.001 >= 3
    for: 15m
    labels: {severity: ticket}
    annotations:
      summary: "Slow error-budget burn — investigate during business hours"
      runbook: "https://runbooks.internal/api-availability"

Alertmanager routing/grouping (sketch):

route:
  group_by: ['alertname', 'service']       # collapse related alerts into one notification
  group_wait: 30s
  group_interval: 5m
  receiver: slack-tickets
  routes:
    - matchers: [severity="page"]
      receiver: pagerduty                   # only pages go to the wake-someone channel
    - matchers: [severity="ticket"]
      receiver: slack-tickets

A page must be actionable and runbook-linked. Before adding any alert, ask the litmus test: user hurt? human must act now? If not both, downgrade it to a ticket or a dashboard panel — never a page.

Hands-on exercises

Reuse your module 06 metrics + Prometheus setup, and add Alertmanager:

# add to prometheus.yml
rule_files: ["alerts.yml"]
alerting:
  alertmanagers:
    - static_configs: [{ targets: ["alertmanager:9093"] }]

1. Define an SLI, SLO, and error budget on paper

For your /orders service, write down: one availability SLI (as a ratio) and one latency SLI; a realistic SLO for each over 30 days; and the resulting error budget as both a percentage and — assuming 1,000,000 requests/30 days — an absolute number of allowed failures.

2. Write a naive threshold alert and feel why it's bad

Write alert: AnyError firing when `rate(http_requests_total{status=~"5.."}[1m])

0`. Generate a single 500. Watch it fire.

Expected: it pages on one error — pure noise, since a single transient 500 is within budget and needs no human. Write down why this exact alert is the seed of alert fatigue, then delete it. (This is the "alert on any error" trap.)

3. Convert it to a burn-rate alert

Replace the naive alert with the fast-burn rule from the command reference (burn rate ≥ 14.4 over 5m and 1h). Generate a sustained high error rate (e.g. a route that 500s 30% of the time under load).

Expected: the alert does not fire for a brief blip, but does fire once the high error rate is sustained across both windows — urgency matched to a genuine fast burn. You've turned a noisy threshold into a symptom-based, budget-aware alert.

4. Prove the two windows kill false positives

With the burn-rate alert active, generate a 20-second spike of errors and stop. Then generate a sustained one.

Expected: the 20-second spike does not page (it can't satisfy the 1h window); the sustained burn does. Write down how the two-window requirement is doing the noise suppression — the short window makes it fast, the long window makes it trustworthy.

5. Grade and route by severity

Add the slow-burn ticket rule and Alertmanager routing so severity=page goes to one receiver and severity=ticket to another (use a webhook/log receiver locally). Trigger both a fast and a slow burn.

Expected: the fast burn routes to the "page" receiver; the slow burn to the "ticket" receiver — the same SLO producing graded responses. This is severity-based routing keeping the paging channel sacred.

6. Group related alerts into one notification

Configure group_by: ['service']. Make three alerts fire for the same service at once (e.g. availability, latency, and a dependency alert from one DB outage).

Expected: Alertmanager sends one grouped notification instead of three separate pages — the anti-fatigue behaviour when one root cause trips many alerts. Remove grouping and observe the three-page spam you just prevented.

7. Write a runbook and link it

Pick your fast-burn alert and write a short runbook: what it means, how to confirm it's real (which dashboard/metric), how to investigate (module 07's metric → trace → logs loop), and first remediation steps. Link it in the alert's annotations.runbook.

Expected: the alert now carries a link that turns "woken up, confused" into "woken up, here's step 1." Note how much of the runbook is just module 07's investigation loop written down.

8. Diagnose and fix: the on-call rotation everyone quit

A team's #alerts channel gets ~400 alerts/day and three engineers have quit on-call in two months. Here's a sample of their alert rules. Diagnose the fatigue sources and rewrite the alerting strategy.

- alert: HighCPU
  expr: cpu_usage > 80
  labels: {severity: page}
- alert: PodRestarted
  expr: increase(pod_restarts[5m]) > 0
  labels: {severity: page}
- alert: AnyServerError
  expr: rate(http_requests_total{status=~"5.."}[1m]) > 0
  labels: {severity: page}
- alert: SlowQuery
  expr: db_query_seconds > 1
  labels: {severity: page}
- alert: DiskWillFillEventually
  expr: disk_used_percent > 70
  labels: {severity: page}

Independent challenge

No code given. Take the fully instrumented service from module 06/07 (RED metrics + traces + correlated logs) and design its complete alerting strategy from scratch, defending every choice against the fatigue test. Specifically: (1) define availability and latency SLIs/SLOs for the service and compute the error budget; (2) write multi-window, multi-burn-rate Prometheus alerts that page on a fast burn and ticket on a slow burn of each SLO — and nothing else pages; (3) enumerate at least three cause signals (CPU, restarts, pool saturation, slow queries) and justify, for each, why it is a dashboard/ticket and not a page; (4) configure Alertmanager to route by severity and group by service; and (5) write a runbook for your fast-burn page that reuses module 07's metric → trace → logs investigation loop. Then prove it: drive a sustained outage and show exactly one grouped, actionable page fires with a working runbook link; drive a brief blip and a high-CPU-but-healthy period and show neither pages. Reach back to module 00: explain how a service that degrades gracefully (fallbacks, circuit breakers) spends its error budget more slowly and therefore pages less — reliability engineering and error-handling are the same discipline viewed from two ends.

Common mistakes & troubleshooting

• Alerting on causes instead of symptoms. CPU, memory, restarts, slow queries page constantly and mostly don't matter to users. Page on symptoms (errors, latency, availability — the SLOs); keep causes as dashboard/diagnostic signals.
• "Alert on any error." A single transient error is within budget and needs no human. Alert on sustained budget burn, not on any single failure.
• Everything is a page. Not every real problem is urgent. Grade severity (page/ticket/info) and route accordingly; keep the paging channel for wake-someone-now, actionable events only.
• Non-actionable pages. If the response is "watch it" or "it self-heals," it should never have paged. Every page must have a concrete action (and a runbook). This one rule removes most fatigue.
• Raw thresholds instead of burn rate. > 1% for 5m is both noisy (blips) and crude (ignores severity). Multi-window burn-rate alerts scale urgency and suppress false positives.
• No grouping/dedup. One outage tripping ten alerts sends ten pages. Group by service/root cause so one incident is one notification.
• No runbooks. An alert with no runbook wakes someone who then doesn't know what to do. Link the meaning + confirmation + first steps.
• Set-and-forget. Alerting rots as the system changes. Review fired alerts regularly and prune/tune the noisy ones — keeping the signal-to-noise ratio high is ongoing work.
• SLO of 100%. Impossible and infinitely costly, and it means zero error budget so everything pages. Pick a realistic SLO that leaves budget to ship.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. What is alert fatigue, why is a noisy alert worse than no alert, and what single ratio is every practice in this module trying to protect?
2. Distinguish symptom-based from cause-based alerting with an example of each, and state the rule for which should page. Where do cause signals still belong?
3. Define SLI, SLO, and error budget, and show how the error budget follows from the SLO with a number.
4. What is burn rate, and why is a multi-window multi-burn-rate alert better than a raw "error rate > 1% for 5m" threshold — on both the false-positive and the severity axes?
5. State the "every page must be actionable" rule and give the litmus-test question you apply to any proposed alert before it's allowed to page.
6. Name three anti-fatigue mechanisms beyond choosing the right metric (think: what Alertmanager does, what a runbook is for, and what severity grading buys you).

Cumulative review

Closed-book. This covers modules 00-08 — the whole track so far. Write each answer before expanding, and if one exposes a gap, go redo that module's exercises rather than just reading the answer.

1. (00+01) A downstream payment provider is timing out. Walk the entire correct handling path: what the code does (fail-fast vs fail-safe vs propagate, fallback/degrade), what the client sees (status + shape), what's logged and at what level, and how the error budget (08) is affected differently depending on whether you degrade gracefully or return a 5xx.
2. (02+03) You're adding a PAYMENT_API_KEY and a PAYMENT_TIMEOUT_SECONDS. Classify each (static/dynamic/sensitive), say where each comes from and its Settings type, and name every mechanism across the track (03, 05) that keeps the key out of logs, traces, and error responses.
3. (04+05+06) The same failing payment call should leave evidence in all three pillars. State precisely what each pillar records for it — the log (level + key fields), the metric (type + name + labels), the trace (span + attributes + status) — and the one shared id that lets you pivot between them.
4. (06) For the payment integration, give the correct metric type for: total charge attempts; charges currently in flight; charge latency distribution; and explain why payment_provider_id is a fine metric label but payment_intent_id is not.
5. (07) A user reports "checkout was slow at 2:04pm." You have all three pillars correlated. Give the exact sequence of pivots — which pillar you start in, what it tells you, what you open next, and how the shared id carries you through to the root cause.
6. (08) Your availability SLO is 99.9% and last night a 40-minute partial outage burned 60% of the monthly error budget in one go. State: whether a page should have fired and why (burn rate), what severity a separate "disk 75% full" condition should have been that same night, and — using the budget — what the team's dev-vs-reliability priority should be for the rest of the month.
7. (00+04+08) Explain the single thread connecting module 00's "don't swallow exceptions," module 04's "don't log normal events at ERROR," and module 08's "don't page on non-actionable alerts." What is the one discipline all three are instances of, and what does violating it cost you?

Further reading & sources

• Google SRE Book — Service Level Objectives - the definitive treatment of SLIs, SLOs, and error budgets this module builds on.
• Google SRE Workbook — Alerting on SLOs - the multi-window, multi-burn-rate alerting pattern in full, with the burn-rate math.
• Google SRE Book — Being On-Call - the human side: alert fatigue, actionable pages, and sustainable rotations.
• Prometheus — Alerting rules - how to express alert conditions as PromQL plus a for: duration.
• Alertmanager documentation - routing, grouping, deduplication, and silencing — the anti-fatigue plumbing.
• My Philosophy on Alerting — Rob Ewaschuk - the original memo behind "page on symptoms, not causes."
• Track 12: Alerting and on-call basics - the same ideas hands-on with Prometheus rules and Alertmanager routing, grouping and silencing.

Next

09-graceful-shutdown — you can now observe and be alerted about a running system. The next module turns to a specific operational moment that quietly breaks reliability if mishandled: shutdown. When a container is told to stop (every deploy, every scale-down), a naive process drops in-flight requests, corrupts work, and fails health checks — burning error budget on routine operations. You'll learn signal handling (SIGTERM), draining in-flight requests, cleaning up connection pools and background tasks, and how readiness vs liveness probes should behave during shutdown.