Capstone Project: Run the Full SRE Loop End to End

Why this matters

Every module in this track added one station of the SRE loop in isolation — the mindset, an SLI/SLO, an error budget, burn-rate alerting, on-call, incident response, postmortems, and capacity/toil management. Real SRE is none of those alone; it's the whole loop turning once around a real service: you define what reliable means and measure it, you get paged the instant the budget burns too fast, you respond to the fire with a process and a runbook, and you close the loop with a blameless postmortem whose action items actually ship. This capstone is where you prove you can run that entire loop yourself, on the exact Prometheus/Grafana/Alertmanager stack you built in track 12 — turning "we observe our system" into "we operate it as a measured, budgeted, engineered discipline." There's no solution given and no quiz. Finishing this is what "you can do SRE" means.

The project

Take a service you can break and fix on your kind cluster — reuse the fully observable app from track 12's capstone if you built it, or any workload exposing request-count and latency-histogram metrics — and run the complete SRE loop on it: define an SLO, alert on its error budget with multi-window multi-burn-rate alerts, respond to a simulated bad deploy as a real incident with a written runbook, and produce a blameless postmortem with tracked action items. Everything installs via Helm and costs nothing; you may optionally reproduce the alerting on an AKS cluster from track 07 to contrast with Azure Monitor, but clean that up with az group delete when done.

The five required deliverables, each drawing on specific modules:

  1. A defined SLO with a real SLI backed by track 12's Prometheus (modules 01-02) — choose an SLI (availability and/or latency) built as a good/valid ratio from your app's actual scraped metrics, set an SLO target and window you can justify from measured current performance, and derive the error budget in a concrete unit. The SLI and budget must be recording rules so the dashboard and alerts run on cheap precomputed series.

  2. A multi-window multi-burn-rate alert that actually fires under a simulated bad deploy (module 03) — at minimum a fast-burn (page, critical) and a slow-burn (ticket, warning) alert with correctly sized window pairs, routed through the exact Alertmanager tree from track 12 module 06. You must force it to fire with a simulated bad deploy (drive 5xx) and watch it travel Inactive → Pending → Firing → routed → delivered — an alert you've never seen fire is a guess.

  3. A written on-call runbook for that alert (module 04) — a genuinely good runbook (a stranger to the service could resolve the incident from it): the alert's user-terms meaning and impact, exact diagnostic steps (dashboard link, specific PromQL/LogQL from track 12's correlation flow, kubectl commands), named likely causes with specific fixes, escalation guidance, and SLO/budget context — stored as runbook-as-code in a repo and linked from the alert's runbook_url.

  4. A full incident response walkthrough, simulated (module 05) — run the bad deploy as a real incident: declare it with a severity from your own rubric, take IC, keep a timestamped timeline separating fact from speculation, mitigate to stop user impact (roll back) before fully understanding root cause, post a stakeholder update on cadence, and resolve when user impact ends. Keep the timeline — deliverable 5 consumes it.

  5. A blameless postmortem with concrete action items (module 06) — from your incident timeline, write a complete postmortem: summary, timeline, impact (including error budget spent, computed in PromQL), systemic root cause(s) via an explicit five-whys chain (not a scapegoat), what went well / where you got lucky, and action items that are every one of specific, owned, dated, and filed as real backlog tickets — with at least one closing the systemic gap that let the bad deploy through.

Acceptance checklist

Work isn't done until you can demonstrate every one of these:

  • An SLI is defined as a good/valid ratio over your app's real scraped metrics (availability and/or latency-under-threshold — not a resource metric, not up), and you can justify every label as bounded.
  • An SLO target + window is set, and you can defend the number from the app's measured performance (achievable, not aspirational) — with the error budget derived in a concrete unit (minutes or requests).
  • The SLI and error-budget expressions exist as recording rules loaded by the Operator (correct release label), and an SLO dashboard in Grafana shows SLI-vs-target, budget remaining, current burn rate, and time-to-exhaustion.
  • A fast-burn alert (critical) and a slow-burn alert (warning) exist with correctly sized window pairs and burn multipliers; each leg of the and can actually cross (you graphed them), and each routes to the intended receiver (verified in the Alertmanager UI or amtool config routes test).
  • You forced the fast-burn alert to fire under a simulated bad deploy and watched the full Inactive → Pending → Firing → routed → delivered transition to an observable receiver — and confirmed the slow-burn stayed silent (it discriminates).
  • A runbook exists as code in a repo, is linked from the alert's runbook_url, and passes the "a stranger could resolve the incident from it" test — exact links, queries, commands, named causes with specific fixes, escalation, SLO/budget context.
  • You ran a full incident: declared with a severity, took IC, kept a timestamped fact-vs-speculation timeline, mitigated first (rolled back) and resolved on user-impact-stopped, with at least one cadenced stakeholder update.
  • A blameless postmortem exists with all sections, reaches a systemic root cause via an explicit five-whys chain (no human scapegoat), quantifies error budget spent in PromQL, and honestly covers "what went well / where we got lucky."
  • Every action item is specific, owned, dated, and filed as a real backlog ticket, with a named follow-up mechanism — and at least one closes the systemic gap that allowed the bad deploy (e.g. a CI validation/rollback guardrail from track 10).
  • At least one diagnose-and-fix from the track is reproducible on your system: a burn-rate alert that never fires because the window is wrong (module 03), an exhausted error budget and the policy conversation it forces (module 02), or a postmortem with vague action items you then made SMART (module 06).
  • If you built on AKS, all billable resources are cleaned up with az group delete.

Hints (not a solution)

  • Sequence it the way the track was ordered. Define the SLI/SLO and record them first (you can't alert or dashboard on a budget you haven't defined), then build the burn-rate alerts and SLO dashboard, then write the runbook, then run the incident, and write the postmortem last — the postmortem is only good if it's about an incident you actually ran.
  • Reuse the track 12 stack — don't rebuild it. kube-prometheus-stack (Prometheus/Grafana/Alertmanager) and, ideally, the fully observable app from track 12's capstone are already on your kind cluster. This capstone is mostly pointing SRE machinery at that stack, not reinstalling it.
  • Set the SLO from measured performance, then make it bite once. Run your SLI over your longest window and set the target at or just below it. An SLO that never triggers a conversation isn't constraining anything — the simulated bad deploy is how you make it bite on purpose.
  • Force the alert; don't wait for it. Simulate the bad deploy (drive 5xx above 14.4× your budget) so you watch the fast-burn fire and route — and graph each leg of the and first so you don't fall into the never-fires window trap (module 03).
  • Mitigate before you understand. In the incident, roll back to stop user impact and resolve on that; the "why did the deploy break it" investigation is the postmortem's job, not the incident's (module 05). Resolving ≠ root-causing.
  • Make the postmortem blameless and its action items real. Push the five-whys past any human-shaped answer to a systemic gap, and make every action item a tracked ticket with an owner and a due date — "improve monitoring" is how the same incident recurs (module 06).
  • The whole thing is the loop, not the parts. Five polished artifacts that don't connect aren't the deliverable — the deliverable is one service where the same SLO drove the alert that fired the incident that produced the postmortem whose action item closes the gap. Depth on the loop closing beats breadth.
  • Don't gold-plate. One SLO run cleanly through the full loop beats three half-run ones. The end-to-end connection — budget → burn alert → incident → postmortem → shipped fix — is the goal.

Next

Before you move on: if any acceptance item is checked only because "it exists," go back and prove it the hard way — show the alert actually firing under the bad deploy and routing to a receiver, the runbook actually followable by someone who's never seen the service, the incident actually run with a real timeline, and the postmortem's action items actually filed as tickets with owners. An SRE loop you haven't seen turn once — budget burning, page firing, incident declared, mitigation applied, postmortem written, fix shipped — is a loop you haven't verified. When every box is genuinely ticked, you've finished the track: you can define reliability formally, get paged only when it matters, run a coordinated incident, and close the loop so the same failure doesn't return.

This track was the operational-maturity layer. Two later tracks build directly on it:

  • 21-cost-management-and-finops applies the exact same budgeting discipline to money that this track applied to reliability and toil. The error budget, the toil budget, and now a cost budget are the same idea — a tracked number that forces a decision when breached — and FinOps is where you learn to right-size, tag, and forecast cloud spend as an ongoing practice, using the same historical-metrics forecasting (predict_linear) you used for capacity planning in module 07.
  • 22-disaster-recovery-and-chaos-engineering takes the incident response, runbooks, and postmortems you just practiced and turns them proactive — deliberately breaking your systems to prove they recover, and designing the backup/failover strategies that your SLOs and error budgets say you need.

You now have the operational foundation both of those build on.

Back to the track index · Back to main curriculum