On-Call in Depth

Why this matters

Track 12 module 06 taught you to make alerts actionable and named alert fatigue as the enemy; this module builds the human system that receives those alerts and formalizes the fix. A great alert that pages a burned-out engineer with no runbook at 3am still ends in a slow, error-prone response. On-call done well — humane rotations, clear escalation, and runbooks that genuinely help — is what turns your burn-rate pages from module 03 into fast, repeatable incident response instead of a lottery on who's awake and what they remember.

Concepts

Rotation design — sustainable, not heroic

An on-call rotation is the schedule deciding who's responsible for responding to pages at any given time. The goal is sustainability: on-call must be a normal part of the job, not a punishment or an act of heroism, or your best people burn out and leave. Practical rules that make a rotation humane: a big enough pool (6-8 engineers so any one person is on-call roughly one week in 6-8, not every other week); follow-the-sun across time zones where you have them, so nobody is routinely paged at 3am local; primary and secondary on-call simultaneously (the secondary is backup if the primary doesn't ack); reasonable shift length (a week is common; longer burns people out, much shorter fragments context); and compensation or time-off-in-lieu so on-call is recognized work, not unpaid availability. A rotation designed for heroics — two people, always on — is a resignation generator.

Escalation policies — what happens when nobody acks

An escalation policy defines what happens to a page that isn't acknowledged. A page fires to the primary on-call; if they don't ack within N minutes (say 5), it escalates to the secondary; if they don't ack either, it escalates further — to the team lead, then an engineering manager, then a broader group. This is the safety net that ensures a page is never silently dropped because the one person it went to is asleep, driving, or has a dead phone. It's the human counterpart to the Alertmanager routing tree from track 12 module 06 (which decided which receiver got the alert); escalation decides what happens when that receiver doesn't respond. Every escalation policy needs a defined ack action (a way for the responder to say "I've got it," stopping the escalation) and a final tier that is guaranteed to reach someone — the escalation must never run off the end into nobody.

Runbooks — the difference between a good one and a useless one

A runbook (or playbook) is the document a paged engineer opens to know what to do about this specific alert. Most runbooks are useless, and the difference is stark. A useless runbook restates the alert ("HighErrorRate means the error rate is high") and offers vague advice ("investigate the issue, check the logs"). A good runbook is written for a stressed engineer at 3am who may not own this service and assumes nothing: it states what the alert means in user terms and its severity/impact; gives the exact first diagnostic steps — the specific Grafana dashboard link, the exact PromQL/LogQL queries to run (reusing your track 12 correlation flow), the specific kubectl commands; lists the most common causes and their specific fixes ("if X, roll back with this command; if Y, scale this deployment"); says when and how to escalate and who owns the service; and links the relevant SLO/error-budget context. The test: could someone who has never seen this service follow it to a resolution? If not, it's prose, not a runbook. The runbook: annotation you added to the burn-rate alert in module 03 should point at exactly this.

Runbook as code — keeping runbooks alive and linked

A runbook that drifts out of date is worse than none — it confidently sends the responder down a path that no longer works. The durable pattern is runbook as code: store runbooks in the same Git repo as the service (track 08), version them alongside the code, review changes in PRs, and link them directly from the alert via the runbook/runbook_url annotation so the paged engineer is one click from the right document — never hunting a wiki at 3am. Some teams go further and make runbook steps executable (scripts or "you can safely run this command" blocks), but the baseline discipline is: runbooks live with the code, are reviewed when the service changes, and every paging alert links to exactly one. This closes the loop with the postmortem process (module 06): a common, high-value action item is "the runbook was wrong/missing — fix it," and if runbooks are code, that's a reviewable PR, not a lost intention.

Alert fatigue, formally fixed — the operational-review loop

Track 12 module 06 named alert fatigue and gave tactical fixes (grouping, symptom-based alerting, severity routing). The formal fix is a standing process: a regular on-call review (often weekly) where the team looks at every page from the last shift and asks, for each: was it actionable? did it require a human? was the runbook useful? If a page was non-actionable or auto-resolved, it gets fixed or deleted — tuned threshold, converted to a ticket, silenced, or removed — and that fix becomes a tracked action item. You also watch a metric: pages per shift. If on-call is being woken more than a small handful of times a night, that's a bug in the alerting, treated with the same seriousness as a bug in the code. This is the systemic version of module 03's burn-rate philosophy: you don't just make individual alerts good, you run a loop that keeps the whole alert set healthy over time — and it feeds directly from the postmortem action items in module 06.

Command reference

On-call is mostly process, but it touches real tooling: Alertmanager grouping/inhibition (track 12 module 06) reduces page volume, and amtool lets you inspect and silence from the CLI during a shift.

amtool during a shift — silence known noise without opening the UI:

amtool --alertmanager.url=http://localhost:9093 silence add \
  alertname=ErrorBudgetSlowBurn service=demo \
  --duration=2h --comment="known slow burn, fix tracked in JIRA-123" --author=oncall
amtool --alertmanager.url=http://localhost:9093 silence query

• silence add alertname=... service=... — label matchers selecting exactly what to mute (the same matcher model as the UI silences in track 12 module 06).
• --duration=2h — time-boxed; the silence auto-expires so muted noise doesn't become permanently invisible (module 06's "silence, don't delete" discipline).
• --comment=... --author=... — mandatory hygiene: a silence with no reason is how alerts vanish forever. Always record why and who.
• silence query — list active silences; run it before you hand off a shift so the next on-call knows what's muted and why.

Hands-on exercises

Continue on the kind cluster with the Alertmanager stack and the burn-rate alerts from module 03. Several exercises are written artifacts (rotation, runbook) — those are the point; on-call quality is judgment, not commands.

1. (paper) Design a humane rotation. For a team of 7 engineers across two time zones, design a rotation: shift length, pool size, primary/secondary, follow-the-sun handoff, and comp/time-off policy. Then compute how often each person is on-call. Contrast with a 2-person always-on rotation and write one sentence on why the latter causes attrition.

2. (paper) Design an escalation policy. Write the escalation chain for a critical page: primary → (N min, no ack) → secondary → (N min) → team lead → manager. Specify the ack timeout, the ack action, and the guaranteed final tier. Then identify the failure mode of a policy whose last tier is a single person who might be unreachable.

3. (WSL2) Wire the escalation intuition into Alertmanager timing. You may not have a full pager tool, but you can model the "re-notify if unacked" behavior with Alertmanager's repeat_interval (track 12 module 06). Set a short repeat_interval on the critical route, force the fast-burn alert from module 03, and observe the receiver get re-notified until you silence/resolve it — the local analog of "escalate if not acked."

4. (paper → repo) Write a good runbook for your fast-burn alert. For the ErrorBudgetFastBurn alert from module 03, write a runbook that a stranger to the service could follow: what it means in user terms, impact/severity, the exact first dashboard link and PromQL/LogQL queries (reuse your track 12 correlation flow), the top 2-3 likely causes with specific fixes (e.g. "if the burn started right after a deploy, roll back with kubectl rollout undo deployment/demo"), when to escalate, and the SLO/budget context. Store it in a repo (track 08) and set the alert's runbook_url to point at it.

5. (paper) Grade a useless runbook. Take this runbook: "HighErrorRate: the error rate is high. Investigate the issue and check the logs. Escalate if needed." List every way it fails the "could a stranger resolve the incident?" test, and rewrite one step of it concretely. This trains you to spot the difference under pressure.

6. (WSL2) Silence hygiene during a shift. Using amtool silence add, silence the slow-burn alert with a duration, comment, and author, then silence query to confirm. Then deliberately create a bad silence (no comment, no expiry analog) and articulate why an un-commented, open-ended silence is how a real alert disappears for the day it matters (track 12 module 06's "silence, don't delete" — now the failure is the silence itself).

7. (paper) Run an on-call review. Given this fake shift log — [12 pages: 4 × NodeCPUHigh (auto-resolved, no action), 3 × PodRestarted (normal), 2 × ErrorBudgetSlowBurn (real, ticketed), 1 × ErrorBudgetFastBurn (real incident), 2 × DiskWarning (acked, nothing done)] — classify each page as keep / tune / delete, propose the specific fix for each non-actionable one, and compute pages-per-shift before and after your changes. This is the formal alert-fatigue fix in action.

8. Diagnose and fix: alert fatigue from a page storm that's really one incident. Reproduce the failure track 12 module 06 warned about, now from the on-call seat: remove group_by (or set it wrong) so a single bad deploy makes 20 pods each fire the fast-burn alert, and the on-call gets 20 pages for one incident — the definition of fatigue. Diagnose: in the Alertmanager UI the 20 alerts are the same incident (same alertname, same service) but arrived as 20 separate notifications because nothing grouped them; the responder can't tell it's one problem. Fix: set group_by: [alertname, service] so the 20 collapse into one page, add an inhibit_rule so the fast-burn critical suppresses the redundant slow-burn warning for the same service, and file the "why did 20 pods break at once" question for the postmortem (module 06). Lesson: on-call health is a systems property — the fix for being paged 20 times isn't to answer faster, it's to make one incident produce one page (grouping + inhibition + the review loop), which is exactly why the weekly review treats a page storm as a bug to fix, not a night to survive.

Independent challenge

Drawing on this module, the Alertmanager grouping/inhibition/silencing from track 12 module 06, the burn-rate alerts from module 03, and the Git workflow from track 08, design a complete on-call system for the demo service and prove one part of it works end to end — no templates given. Produce: a humane rotation (with the math on how often each person is paged), an escalation policy with a guaranteed final tier, a genuinely good runbook-as-code for your fast-burn alert (stored in a repo and linked via runbook_url), and a written weekly-review procedure that would keep pages-per-shift low over time. Then demonstrate the alert-fatigue fix live: cause a multi-instance incident, show it producing a page storm without grouping, and show it collapsing to a single page with grouping + inhibition. The deliverable is the system plus the demonstrated fix — a rotation and runbook on paper that you've never proven reduce noise are still hope, not strategy (module 00).

Common mistakes & troubleshooting

• Heroic rotations. Two people always on-call burns out your best engineers. Build a pool of 6-8, primary+secondary, reasonable shifts, and recognized comp.
• Escalation that runs off the end. A policy whose last tier is one possibly-unreachable person can drop a page silently. Guarantee a final tier that always reaches someone.
• No ack mechanism. Without an explicit ack that stops escalation, either the page escalates when it shouldn't or nobody knows it's being handled.
• Useless runbooks. "Investigate the issue, check the logs" helps no one at 3am. A runbook must be followable by a stranger: exact links, queries, commands, named causes and fixes, and escalation guidance.
• Runbooks that drift. A stale runbook confidently misleads. Keep runbooks as code, reviewed in PRs alongside the service, and linked from the alert.
• Open-ended, un-commented silences. A silence with no expiry and no reason is how a real alert vanishes. Always time-box, comment, and record the author.
• No on-call review loop. If nobody audits pages, alert fatigue accrues forever. Run a weekly review, treat non-actionable pages as bugs, track pages-per-shift.
• Answering a page storm faster instead of fixing it. 20 pages for one incident is a grouping/inhibition bug, not a test of stamina.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

Next

05-incident-response-process — the page reached a rested on-call engineer with a good runbook. Now formalize what happens next when it's a real incident: severity levels, the incident commander role, communication while the fire is burning, and how you declare and resolve.