Platform Observability and SLOs

Why this matters

The moment other teams depend on your platform, the platform is production — and if the scaffolder, the portal, or the provisioning pipeline is down or slow, you don't block one team, you block all of them. Yet platform teams routinely monitor the workloads they host meticulously while flying blind on their own services. This module fixes that by turning the SRE discipline of track 20 inward: the platform gets its own SLIs, SLOs, error budgets, and burn-rate alerts, with your internal developers as the customers whose experience those SLOs protect. This is platform-as-a-product (module 00) made operationally real — you don't just say developers are customers, you get paged when you fail them.

Concepts

The platform is a production service to its customers

Every argument from track 20 applies to the platform, only the "users" are internal engineers. When a developer clicks "create a new service" and the scaffolder times out, that's an outage — to them, indistinguishable from any other production failure, and it stops their work. When a self-service database request (module 03) hangs for twenty minutes, that's a latency SLO breach. The platform's components — the portal, the scaffolder, the provisioning pipeline, the CI templates, the golden-path controller — are services, and services that many teams depend on are critical services.

This reframing is the operational half of platform-as-a-product. Module 00 said developers are customers; this module says: then measure whether you're keeping your promises to them, the same rigorous way track 20 taught you to measure promises to end users. A platform team that doesn't know its own scaffolder's success rate is exactly the "hope is not a strategy" failure from track 20 module 00 — it's up because nothing's broken yet.

Platform SLIs — what "working" means for a developer

Defining an SLI (track 20 module 01) means picking a good-events/valid-events ratio that reflects what the customer experiences. For the platform, the customer is a developer, so the SLIs are about their interactions:

Scaffolder success rate — fraction of "create new service" requests that succeed end to end (repo created, CI wired, catalog registered). A developer who gets a half-scaffolded service experienced a failure even if no component crashed.
Provisioning latency — fraction of self-service infra requests (module 03) that complete within a target (e.g. 95% under 5 minutes). Slow self-service is a broken promise even when it eventually succeeds.
Portal availability — fraction of portal page loads / catalog queries that succeed. If developers can't find or act on their services, the front door is down.
Golden-path deploy success — fraction of deploys through the platform's path (track 10 GitOps) that reach Healthy. This measures the paved road's own reliability, not the app's.

Notice these are all built on the exact instrumentation you learned in track 12 — the platform's own services export metrics, get scraped by Prometheus, and traces flow through OTel, just like any workload. The platform monitors itself with its own product.

Platform SLOs and error budgets — with internal customers

Once you have SLIs, you set SLOs — targets over a window — and derive an error budget (track 20 module 02), exactly as before, but now the budget governs platform decisions. If the scaffolder's SLO is 99.5% success and you've burned the month's budget, that's the signal to freeze risky platform changes and stabilize — the same velocity-vs-reliability tradeoff, now applied to the platform team's own shipping. The error budget makes "should we ship this risky scaffolder refactor this week?" a number, not an argument.

Setting the right target matters as much as before: 100% is still the wrong target (module 00 of both this track and track 20), and an SLO so loose it never bites doesn't constrain anything. But there's a platform-specific nuance: because the platform is a dependency of every team on it, its SLO effectively caps the reliability those teams can offer their users — a team can't run a 99.9% service on a 99% platform without absorbing the difference. The platform's SLO is a promise other teams build their own promises on, which is a strong argument for setting it deliberately high on the critical paths (portal read, deploy pipeline) and being honest about the rest.

Burn-rate alerts — paging the platform team on customer pain

The alerting discipline from track 20 module 03 transfers directly: multi-window, multi-burn-rate alerts on the platform's error budget, routed through the exact Alertmanager tree from track 12 module 06. A fast-burn alert on scaffolder success rate pages the platform on-call; a slow-burn alert files a ticket. The rule from track 12 still governs: alert on symptoms the customer feels (scaffolds failing, provisioning hanging), not on causes (a pod restarted) that no developer noticed.

This is where "developers are our customers" stops being a slogan and becomes a 2am page. The platform team goes on-call for its internal customers — runs incident response (track 20 module 05) when the golden path is broken, writes blameless postmortems (track 20 module 06) when a scaffolder change broke every new service for six hours, and tracks action items to real backlog work. The whole SRE loop from track 20's capstone runs again, this time with the platform itself as the service.

Observability of the platform vs. provided by the platform

There's a subtle duality worth naming. The platform provides observability to app teams (the golden path pre-wires OTel and a dashboard from track 12 — module 01), and the platform has observability of itself (this module). These are different concerns with the same tooling. A common failure is doing the first and forgetting the second: the platform lovingly instruments every hosted app while its own scaffolder emits no metrics at all.

A related duality is whose SLO is whose. When a developer's service is slow, is it the app's SLO breaching or the platform's? The platform is responsible for the paved road (the deploy pipeline reached Healthy, the mesh routed the request, the node had capacity — tracks 07/13/23) but not for the app team's business logic. Drawing that line clearly — in dashboards, in on-call scope, in postmortems — is essential, or every app incident becomes a platform page and the platform team burns out owning problems that aren't theirs. The platform's SLIs must measure the platform's contribution, not the aggregate of everything running on it.

Command reference

The tooling is exactly track 12's Prometheus/Grafana/Alertmanager and track 20's SLO machinery, now pointed at the platform's own services. Nothing new to install.

Command	What it does	From
`curl -s localhost:8080/metrics`	Confirm the platform's own service exports metrics	track 12
`promtool query instant <expr>`	Test an SLI expression against the platform's metrics	track 12/20
`promtool check rules platform-slo.yaml`	Validate the platform's recording/alerting rules	track 20
`amtool config routes test`	Verify a platform burn-rate alert routes to the right receiver	track 12
`argocd app list -o json`	Source data for a golden-path deploy-success SLI	track 10

Multi-part examples (the platform's own SLI and alert, as PromQL — know each part):

A scaffolder success-rate SLI as a recording rule (a good/valid ratio, track 20):

# platform-slo.yaml — the platform's OWN SLO, monitored with its OWN product
groups:
- name: platform-scaffolder-slo
  rules:
  - record: platform:scaffolder_success:ratio_rate5m
    # good = completed scaffolds; valid = all scaffold attempts (track 20 module 01)
    expr: |
      sum(rate(scaffolder_runs_total{status="success"}[5m]))
      /
      sum(rate(scaffolder_runs_total[5m]))

A fast-burn alert on that SLO (multi-burn-rate, track 20 module 03):

  - alert: ScaffolderErrorBudgetFastBurn
    # firing means new services are failing to scaffold NOW — a customer-felt symptom
    expr: |
      (1 - platform:scaffolder_success:ratio_rate5m) > (14.4 * 0.005)
      and
      (1 - platform:scaffolder_success:ratio_rate1h) > (14.4 * 0.005)
    for: 2m
    labels: { severity: critical }          # pages the platform on-call
    annotations:
      summary: "Scaffolder burning error budget fast — new services are failing"
      runbook_url: "https://backstage.acme.io/docs/platform/runbooks/scaffolder"

Part	Meaning
`14.4 * 0.005`	14.4× burn against a 0.5% budget (99.5% SLO) — the fast-burn threshold from track 20 module 03
two-window `and`	requires both the 5m and 1h windows to breach, so a brief blip doesn't page (track 20)
`severity: critical`	routes to the pager via track 12's Alertmanager tree; `warning` would file a ticket
`runbook_url`	links the runbook (track 20 module 04) — a platform alert needs a runbook like any other

Hands-on exercises

You need the track 12 Prometheus/Grafana/Alertmanager stack on your kind cluster and a platform service to instrument (the Backstage from module 02, a scaffolder script, or any stand-in that emits metrics).

Instrument a platform service. Take your scaffolder (or a stand-in script) and make it emit a scaffolder_runs_total{status=...} counter — success and failure. Confirm curl localhost:<port>/metrics shows it and Prometheus scrapes it. The platform now monitors itself with track 12's own tooling.
Define a platform SLI. Write the scaffolder success-rate SLI as a good/valid ratio (track 20 module 01) and test it with promtool query instant. Justify why "success" is measured end to end (repo + CI + catalog), not "the process didn't crash" — a half-scaffold is a customer-felt failure.
Set a platform SLO and derive its budget. Choose an SLO target for the scaffolder you can justify from measured performance (track 20 module 02), and compute the error budget in a concrete unit (failed scaffolds per month). Write one sentence on what "spending" this budget looks like (a bad scaffolder deploy that broke creates for an afternoon).
Record the SLI + budget as rules. Load platform-slo.yaml recording rules into your Prometheus (correct release label, track 20 module 03) and build a Grafana panel showing scaffolder SLI-vs-target and budget remaining. The platform now has an SLO dashboard for itself.
Add multi-burn-rate alerts and force one to fire. Add the fast-burn (critical) and slow-burn (warning) alerts. Then break the scaffolder on purpose (make it fail a fraction of runs) and watch the fast-burn go Inactive → Pending → Firing → routed through your Alertmanager tree (track 12 module 06), while the slow-burn stays silent. An alert you've never seen fire is a guess (track 20 module 03).
Write a platform runbook. For the scaffolder alert, write a runbook (track 20 module 04) a stranger to the platform could follow: what the alert means to a developer, dashboard link, the exact queries/commands to diagnose (is it GitHub API rate limits? the catalog registration step? the template render?), likely causes with fixes, and escalation. Link it from the alert's runbook_url and store it in TechDocs (module 02).
Draw the ownership line. For a scenario where a developer's service is slow, write down which parts are the platform's SLO (deploy reached Healthy, mesh routed, node capacity — tracks 07/13/23) and which are the app team's (their business logic, their query). Then design the dashboard/on-call scope so an app-logic problem does not page the platform team. This line is what keeps the platform from owning everyone's incidents.
Diagnose-and-fix: the platform that monitors everyone but itself. You inherit a platform whose golden path beautifully instruments every hosted app (OTel, dashboards, SLOs — module 01) but whose own scaffolder and portal emit zero metrics; the team finds out the scaffolder is down only when developers complain in Slack. Reproduce the gap (a platform service with no metrics and no SLO). Then fix it: instrument the service, define its SLI/SLO, add a burn-rate alert that would have paged before the Slack complaints, and — importantly — write down why the team fell into this (they treated observability as something the platform provides to others, not something it has of itself). The lesson is the duality: providing observability and having observability are different jobs, and skipping the second is how a platform blindsides its own customers.

Independent challenge

Drawing on this module and tracks 12 and 20 (and the SRE capstone, 20-sre-practices/08-capstone-project), run the complete SRE loop on one platform service of your choice (the scaffolder, the provisioning pipeline, or the portal): define a customer-centric SLI and a justified SLO with an error budget, build multi-burn-rate alerts routed through Alertmanager, force a realistic incident (break the service so the fast-burn fires), run it as a real incident with a timeline and a mitigation (roll back the bad platform change first, understand later), and write a blameless postmortem whose action items close the systemic gap — with at least one action item being a guardrail in the platform's own CI/golden-path (tracks 10/11) so the same platform change can't break every new service again. The deliverable is the loop turning once, with the platform itself as the service and internal developers as the customers — proving you can operate the platform to the same standard you'd hold any production service.

Stuck? One hint

This is track 20's capstone re-run with one swap: the "service" is now a piece of the platform, and the "users" are your fellow engineers. Everything else is identical, so lean on what you already did there — the hardest platform-specific judgment is choosing the SLI so it measures the developer's experience, not your process health. "The scaffolder process had 99.9% uptime" is a vanity metric if 30% of scaffolds silently produced a half-wired repo; the real SLI is end-to-end success from the developer's point of view. And for the postmortem's action item, remember the highest-leverage fix for a platform is almost always a guardrail in the golden path or its CI (tracks 10/11), because that one fix protects every future service the path produces — the same leverage that makes the golden path powerful makes a bug in it catastrophic.

Common mistakes & troubleshooting

Monitoring hosted apps but not the platform. The classic gap: the golden path instruments every app while the scaffolder/portal emit nothing. Providing observability (module 01) and having observability (this module) are different jobs; do both.
Vanity SLIs (process health, not customer experience). "The scaffolder was up 99.9%" while a third of scaffolds produce broken repos is measuring the wrong thing. Measure end-to-end success from the developer's point of view (track 20).
Owning everyone's incidents. If every app-logic problem pages the platform team, the ownership line is blurred and the team burns out. The platform's SLIs measure the platform's contribution (deploy, routing, capacity), not the apps' business logic.
An SLO with no teeth (or 100%). A platform SLO so loose it never bites constrains nothing; 100% is unachievable and forbids ever shipping. Set it from measured performance so the error budget can actually gate risky platform changes (track 20 module 02).
Alerts on causes, not symptoms. Paging on "a scaffolder pod restarted" that no developer noticed is noise; page on "scaffolds are failing" that they feel (track 12 module 06).
No runbook for platform alerts. A platform page with no runbook is as bad as any other; platform on-call needs runbooks (track 20 module 04), ideally in TechDocs (module 02) so they're findable.
Forgetting the platform SLO caps its customers' SLOs. A team can't run a 99.9% service on a 99% platform. The platform's reliability is a floor others build on — set the critical-path SLOs deliberately.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

Show questions

Why is the platform itself a "production service," and who are its users?
Give two examples of platform SLIs and explain why each measures the developer's experience rather than process health.
How does the error budget govern the platform team's own decisions?
Why does the platform's SLO effectively cap the SLOs its customer teams can offer their users?
Explain the duality between observability the platform provides and observability the platform has — and the common failure it produces.
When a developer's hosted service is slow, how do you decide whether it's the platform's SLO breaching or the app team's — and why does the line matter?
Which parts of the track 20 SRE loop apply to the platform, and what's the one thing that changes?

Show answers

Because many teams depend on it: when the scaffolder, portal, or provisioning pipeline is down or slow, it blocks all of them, indistinguishable from any other outage. Its users are the internal engineers (platform-as-a-product made operational).
E.g. scaffolder end-to-end success rate and self-service provisioning latency. Each measures what the developer experiences — a half-scaffolded repo or a 20-minute hang is a failure they feel — rather than whether a process stayed up, which they may never notice.
It gates the platform team's own shipping: if the platform's error budget is spent, that's the signal to freeze risky platform changes and stabilize — the same velocity-vs-reliability tradeoff (track 20 module 02) turned inward.
Because the platform is a dependency of every team on it, so its reliability is a ceiling: a team can't run a 99.9% service on a 99% platform without absorbing the difference. The platform's SLO is a floor others build their promises on.
The platform provides observability to app teams (the golden path pre-wires OTel/dashboards — module 01) and must have observability of itself (its own SLIs/SLOs). Same tooling, different jobs. The common failure is doing the first and forgetting the second — instrumenting every hosted app while the scaffolder emits nothing.
Attribute to the platform only what the platform is responsible for — deploy reached Healthy, mesh routing, node capacity (tracks 07/13/23) — and to the app team their business logic/queries. The line matters because without it every app incident pages the platform team, which owns problems that aren't theirs and burns out.
All of it — SLIs/SLOs, error budgets, burn-rate alerts, on-call, incident response, blameless postmortems, action items. The one thing that changes: the "service" is a piece of the platform and the "users" are internal engineers.

Cumulative review

Closed book. Cover the answers and write each out before checking. These mix the whole track so far — concepts (00), golden paths (01), portals (02), self-service infra (03), abstractions (04), and platform SLOs (05) — and the earlier tracks they compose.

A director asks "isn't this just DevOps with extra steps?" Give the crisp distinction and name what platform engineering adds on top of DevOps, SRE, and infra. (module 00)
Walk the full path of a developer clicking "Create → New HTTP Service": name each artifact produced and the track it comes from. (modules 01-02, tracks 02/09/10/11/12/20/21)
A self-service request for a public-endpoint database is rejected with a raw Azure Policy JSON error. Diagnose the DX failure and rewrite the rejection as it should be. (module 03)
Your service.yaml abstraction has a field for nodeAffinity. What's wrong with that, and what's the principle it violates? (module 04)
Distinguish "too little" from "too much" abstraction, give the tell for each, and explain why the law of leaky abstractions makes over-abstraction hurt. (module 04)
Write, in PromQL, a scaffolder success-rate SLI as a good/valid ratio, and say why "the process was up 99.9%" is the wrong SLI. (module 05, track 20)
Why does the platform need its own SLOs, and how does its SLO relate to the SLOs its customer teams can offer their users? (module 05, track 20)
A guardrail (track 17) blocks a legitimate need. Describe the exception path that keeps the guardrail meaningful without becoming a wall — or a loophole. (module 03, track 17)
Your golden-path template improves from v1 to v2, and your service.yaml abstraction goes from v1 to v2. Explain the shared problem across both and the options to solve it. (modules 01, 04)
Draw the ownership line: a hosted app is slow — which parts are the platform's responsibility (and SLO) and which are the app team's, and why does drawing this line matter? (module 05, tracks 07/13/23)