Track 20: SRE Practices
You can already build systems (Docker, Kubernetes, AKS), ship them (CI/CD and GitOps), and observe them deeply — in 12-observability-deep-dive you stood up Prometheus, Grafana, Loki, and OpenTelemetry yourself and even wrote your first Alertmanager routing and on-call hygiene in module 06. This track is the layer on top of all of that: operational maturity. It formalizes the full Site Reliability Engineering discipline — defining and measuring reliability with SLIs/SLOs, turning the gap to 100% into a spendable error budget, alerting on that budget's burn rate instead of static thresholds, running real incident response, and closing the loop with blameless postmortems and toil reduction.
Where track 12 gave you the signals, this track gives you the discipline that decides how reliable a service should be, proves whether it is, and coordinates what happens when it isn't. The alerting/on-call basics from track 12 module 06 are the baseline this track assumes and formalizes — "alert on symptoms, not causes" becomes SLO-based alerting; "don't burn out on-call" becomes rotation design, escalation, and a measured fix for alert fatigue.
Cost note: like track 12, almost all of this track runs on your local kind cluster — the same
kube-prometheus-stack(Prometheus/Grafana/ Alertmanager) you already installed, now driving SLOs and burn-rate alerts, costs nothing. A couple of exercises optionally point the same alerting at an AKS cluster to contrast with Azure Monitor; if you spin up AKS for those, clean it up withaz group deletewhen done, exactly as track 07 drilled into you.
How this track works
- Go in order — module 00 sets the mindset, 01-02 make reliability measurable (SLIs/SLOs then error budgets), 03 is the technical payoff (burn-rate alerting on track 12's stack), 04-05 are the human response system (on-call then incident response), 06 closes the loop (blameless postmortems), and 07 keeps you ahead of failure (capacity and toil). The capstone runs the whole loop once, end to end.
- Every module (except this index and the capstone) follows the same shape: Why this matters → Concepts → Command reference → Hands-on exercises → Independent challenge → Common mistakes & troubleshooting → Checkpoint quiz → Next. Two modules (02 and 05) also carry a Cumulative review.
- Exercises build on the real Prometheus/Grafana/Alertmanager stack from
track 12: real SLI PromQL, real recording rules, real multi-window
multi-burn-rate alerts that fire under a simulated bad deploy, and real
amtoolrouting checks. Several modules include a deliberate diagnose-and-fix — a burn-rate alert that never fires because the window is wrong, an already-exhausted error budget and the policy conversation it forces, a page storm that's really one incident, an unmanaged incident with no IC, and a postmortem full of vague non-actionable action items. - Module 08 is a capstone with no quiz or challenge — it asks you to run the entire SRE loop on one service: a defined SLO, a burn-rate alert that actually fires under a bad deploy, a written runbook, a full simulated incident response, and a blameless postmortem with concrete action items.
Modules
| # | Module | What it covers | Rough time |
|---|---|---|---|
| 00 | sre-concepts-and-the-reliability-mindset | What SRE is as a discipline, dev velocity vs. reliability, "hope is not a strategy," toil vs. engineering work, the SRE loop | 45-60 min |
| 01 | slis-and-slos | Choosing a good SLI (availability/latency/error rate) as a good/valid ratio, SLA vs. SLO, picking a target that's neither too loose nor too tight | 75-90 min |
| 02 | error-budgets | Computing an error budget from an SLO, spending it, error-budget policies, using it for a real ship-vs-slow-down decision | 75-90 min |
| 03 | slo-dashboards-and-burn-rate-alerts | Multi-window multi-burn-rate alerting on track 12's Prometheus/Grafana/Alertmanager — the payoff of that track | 90-120 min |
| 04 | on-call-in-depth | Rotation design, escalation policies, good vs. useless runbooks (runbook-as-code), a formal fix for alert fatigue | 75-90 min |
| 05 | incident-response-process | Severity levels, the incident commander role, communication during an incident, declaring and resolving | 75-90 min |
| 06 | blameless-postmortems | Writing one, five-whys root-cause, turning action items into real backlog work, why blame kills the practice | 75-90 min |
| 07 | capacity-planning-and-toil-reduction | Forecasting capacity from track 12's historical metrics (predict_linear), identifying and automating toil, the toil budget |
75-90 min |
| 08 | capstone-project | End-to-end: a defined SLO, a burn-rate alert that fires under a bad deploy, a runbook, a full incident walkthrough, and a blameless postmortem | 4-6 hours |
Prerequisites
- Everything from
12-observability-deep-dive: comfortable
with Prometheus, PromQL (
rate, histograms,histogram_quantile, recording and alerting rules), Grafana dashboards, and especially the Alertmanager routing/grouping/silencing and on-call basics from its module 06. This track drives SLOs and burn-rate alerts on exactly that stack. - Everything from 07-aks: comfortable operating a real
cluster with
kubectl, Deployments/rollouts (for mitigation viakubectl rollout undo), and the Cluster Autoscaler/HPA (module 07's capacity planning sets the limits those scale within). - Useful from 10-cicd-and-gitops: CI/CD and GitOps — the error budget governs release decisions here, and postmortem action items land as pipeline guardrails and tracked backlog work.
- An Azure subscription is optional — only the AKS-contrast exercises use it; the whole track runs on the local kind cluster.
Start here → 00-sre-concepts-and-the-reliability-mindset/README.md