SLIs and SLOs

Why this matters

An SLI is the bridge between "the mindset" from module 00 and anything you can actually put on a dashboard or alert on — it's a number that says how well users are being served, and it's built from the exact request-count and latency-histogram metrics you learned to query in track 12 module 02. An SLO is the target you hold that number to. Pick a bad SLI and you'll measure something users don't feel; pick a bad SLO target and you'll either page constantly or never notice real pain. This module is where reliability stops being a vibe and becomes a ratio with a threshold.

Concepts

SLI — a good-events-over-valid-events ratio

A Service Level Indicator (SLI) is a carefully chosen measure of the service level users experience, almost always expressed as a ratio of good events to valid events, in the range 0-100%. "Good" and "valid" are decisions you make: for availability, a good event is a request that returned a non-5xx status, and valid events are all requests you're responsible for; for latency, a good event is a request served faster than some threshold. Framing every SLI as good / valid (rather than raw counts) makes it scale-independent — 99% good is 99% good whether you served a thousand requests or a billion — which is the same reason a ratio beat a raw 5xx count for alerting back in track 12 module 02. The three SLIs you'll use constantly are availability (fraction of requests that succeeded), latency (fraction served under a threshold), and error rate (its complement).

Choosing a good SLI — measure what the user feels

Not every measurable thing is a good SLI. The test is: does this number move when, and only when, users are actually hurt? CPU utilization is measurable and a terrible SLI — it can be at 90% with every user perfectly happy, or at 20% while the service is throwing 500s. Good SLIs are the same symptoms track 12 module 06 told you to alert on: request success ratio, request latency, correctness, freshness of data, throughput where users notice it. A good SLI is also measured as close to the user as you can get — at the load balancer or gateway (the Application Gateway/ingress from track 05 and track 07) is better than deep inside one microservice, because that's closer to what the user actually experienced. And it should be a small set — one or two SLIs per service that genuinely capture user happiness beat a dashboard of twenty.

The latency SLI is a threshold, not an average

Latency deserves special care because the naive version is wrong. "Average latency < 300ms" is a bad SLI: an average hides the tail, and the tail is exactly where users suffer — a handful of 10-second requests vanish into a good-looking mean. The correct latency SLI is a threshold ratio: the fraction of requests served faster than X. You compute it straight from the histogram buckets you met in track 12 — the count of requests in buckets ≤ X divided by the total count. This reuses histogram_quantile's cousin: instead of "what latency is the 95th percentile" you ask "what percentile is 300ms," which is a good/valid ratio. So a latency SLO reads "95% of requests complete under 300ms," never "average latency under 300ms."

SLO — your internal target, and the window it's measured over

A Service Level Objective (SLO) is the target value you set for an SLI over a time window — for example, "availability SLI ≥ 99.9% over a rolling 30 days." Two parts matter equally. The target (99.9%) sets how good is good enough. The window (rolling 30 days) sets over what span you judge it — a rolling window means you're always looking at the last 30 days, so a bad hour ages out gradually rather than resetting at a calendar boundary. The SLO is internal: it's the number your team commits to and drives alerting and the error budget (module 02) from. It is deliberately set tighter than any promise you make to customers, so you get warned and can react before you break an external commitment.

SLA vs. SLO — the promise vs. the target

These are constantly confused. An SLA (Service Level Agreement) is a contract with a customer that includes consequences — refunds, credits, penalties — if you miss it. An SLO (Service Level Objective) is your internal target with no contractual penalty, just an error-budget policy (module 02). The rule is: your SLO should be stricter than your SLA, with margin between them. If your SLA promises 99.9% and your internal SLO is also 99.9%, then the moment you breach your SLO you've already breached the contract — no warning, no room to react. Set the SLO to, say, 99.95% so that burning through your internal budget alarms you well before the customer-facing promise is at risk. Many services have an SLO and no SLA at all (internal services, free tiers); fewer have an SLA without a stricter SLO behind it — and that's a mistake.

Picking a target that's neither too loose nor too tight

The hardest part is choosing the number. Too tight (99.99% on a service that realistically does 99.5%) and you'll blow the budget constantly, page people endlessly, and the SLO becomes a joke everyone ignores — the alert-fatigue failure from track 12 module 06, now at the objective level. Too loose (99% on a payments API) and you'll be "meeting SLO" while users are genuinely suffering, because the target permits pain they feel. Three practical anchors: (1) measure your current performance first — an SLO should be achievable with your current architecture, usually set at or just below where you already are, then tightened over time; (2) let user tolerance set the floor — how bad can it get before users leave or complain?; (3) remember every nine costs exponentially (module 00's table) — don't buy a nine users can't perceive. A good SLO is one you mostly meet but that occasionally bites, because an SLO that never triggers a conversation isn't constraining anything.

Command reference

These build directly on the PromQL from track 12 module 02. The pattern for every SLI is sum(rate(good[w])) / sum(rate(valid[w])).

Availability SLI — the canonical expression, flag by flag:

sum(rate(http_requests_total{code!~"5.."}[30d]))
  /
sum(rate(http_requests_total[30d]))

• http_requests_total{code!~"5.."} — the good events: every request whose status code does not match the regex 5.. (i.e. not a 5xx). !~ is the negated-regex matcher; 5.. matches any three-digit code starting with 5.
• rate(...[30d]) — per-second rate over the SLO window. Over a long window this is effectively "average success rate over 30 days."
• outer sum(...) — collapses all label dimensions to one number (the service-wide ratio). Add by (service) if you want it per service.
• the division — good / valid; the result is the SLI, a number between 0 and 1 you compare against your SLO target (e.g. ≥ 0.999).

Latency SLI — fraction of requests under a 300ms threshold:

sum(rate(http_request_duration_seconds_bucket{le="0.3"}[30d]))
  /
sum(rate(http_request_duration_seconds_count[30d]))

• ..._bucket{le="0.3"} — the histogram's cumulative count of requests that completed in ≤ 0.3s (Prometheus histogram buckets are cumulative — reuse of the le label from track 12 module 02).
• ..._count — the total number of observed requests (the denominator).
• ratio — the fraction served under 300ms; that is the latency SLI. Note it's a threshold ratio, not an average or a single histogram_quantile value.

Hands-on exercises

Reconnect to the kind cluster running kube-prometheus-stack from track 12. You'll write PromQL SLIs against real scraped metrics from the demo app. If the demo app is gone, redeploy any workload that exposes request-count and duration-histogram metrics (the track 12 module 01/02 app).

1. (WSL2) Write an availability SLI against real metrics. Port-forward Prometheus (svc/kps-kube-prometheus-stack-prometheus 9090) and, in the graph UI, write the good/valid availability expression for the demo app over [5m]. Confirm it returns a value near 1.0. Now break something (scale the app to return some 500s, or hit an error endpoint) and watch the SLI drop. You just measured availability as a ratio, not a guess.

2. (WSL2) Write a latency SLI as a threshold ratio. Using the app's *_duration_seconds_bucket, write the fraction-under-300ms expression. Then write the wrong version — histogram_quantile(0.95, ...) compared to a number, and an average (..._sum / ..._count) — and write one sentence on why the threshold-ratio is the correct SLI and the average is misleading (the tail).

3. (WSL2) Pick good vs. bad SLIs for the demo app. List five metrics the app exposes. For each, decide: is this a good SLI (moves when and only when users are hurt) or a bad one (a cause/resource metric)? You should end with request success ratio and latency as good, and CPU/memory/goroutine-count as bad. Write the one-line justification per item — this is the module 00 symptom-vs- cause test applied to SLI selection.

4. (paper) Translate three services into SLI + SLO targets. For (a) a payments API, (b) an internal batch-report generator, (c) a marketing website, write a sensible SLI and an SLO target + window for each, and justify the number. You should end up with very different targets (payments tight, marketing loose, batch measured on freshness/completion not latency) — the point is that the right SLO is service-specific, not a universal 99.9%.

5. (WSL2) Measure current performance, then set the SLO from it. Compute the demo app's actual availability SLI over the longest window you have data for. Then set an SLO target — at or just below what you measured — and write why setting it above your measured performance (aspirationally) would just blow the budget from day one. This is the "achievable first, tighten later" anchor.

6. (paper) SLA vs. SLO with margin. A customer contract (SLA) promises 99.9% monthly availability with service credits if missed. Choose an internal SLO target that gives you warning margin, and explain in two sentences what happens on the day you breach the SLO vs. the day you breach the SLA. Then describe the failure mode of setting SLO = SLA (no warning, contract already broken when your own alarm fires).

7. Diagnose and fix: the vanity SLI that's always green. Someone set the service's "reliability SLI" to up (the Prometheus target-health metric from track 12 module 01) — so the dashboard shows 100% and the SLO is "never breached," yet users have been complaining about 500s all week. Reproduce it: graph avg_over_time(up{job="<app>"}[1h]) and see a flat 1.0 while your real availability SLI from exercise 1 is visibly dropping under injected errors. Diagnose why up is the wrong SLI (it only says "Prometheus could scrape the target," i.e. the process is alive — it says nothing about whether requests are succeeding). Fix by replacing the SLI with the good/valid request-success ratio. Lesson: an SLI that's always green while users suffer is worse than none — it manufactures false confidence. This is the objective-level version of track 12 module 06's "firing ≠ paged" trap: green ≠ users happy unless the SLI measures what users feel.

8. (WSL2) Turn your SLI into a recording rule. SLI expressions over [30d] are expensive to evaluate on every dashboard refresh. Using the recording-rule technique from track 12 module 02, write a PrometheusRule that records the short-window good and valid rates (e.g. job:sli_good:rate5m, job:sli_valid:rate5m) so module 03 can build burn-rate alerts on cheap precomputed series. Confirm the recorded series appear in Prometheus (remember the release label so the Operator loads the rule — the module 01/02 trap).

Independent challenge

Using this module plus the RED-shaped dashboard from track 12 module 03 and the symptom-vs-cause judgment from module 00, define a complete SLI/SLO specification for the track 12 demo app — no expressions given. Choose exactly two SLIs (you'll almost certainly land on availability and latency), write the precise good/valid definition for each in words and as PromQL against the app's real metrics, pick an SLO target and window for each that you can justify from the app's measured current performance (not an aspiration), and state one SLA you could offer a customer that sits looser than your SLO with margin to spare. The written justification — why these two SLIs, why these targets, why this margin — is the deliverable, because choosing what to hold yourself to is the actual SRE skill; the PromQL is the easy part.

Common mistakes & troubleshooting

• Averaging latency. "Average latency < 300ms" hides the tail where users actually suffer. Use a threshold ratio — fraction of requests under 300ms — built from histogram buckets.
• Resource metrics as SLIs. CPU/memory/queue-depth are causes, not what users feel. An SLI must move when and only when users are hurt. (up as an SLI — exercise 7 — is the classic version of this.)
• SLO = SLA. If your internal target equals your contractual promise, you get no warning — the day your own alarm fires, the contract is already broken. Keep the SLO stricter, with margin.
• Aspirational SLOs. Setting the target above your measured performance blows the budget from day one and trains everyone to ignore it. Set it achievable, then tighten.
• Too many SLIs. Twenty SLIs is a dashboard, not a target. One or two that genuinely capture user happiness are what you defend and alert on.
• Measuring far from the user. An SLI deep inside one service can be green while the edge the user hit is red. Measure as close to the user (LB/ingress) as you can.
• Forgetting the window. "99.9% availability" is meaningless without "over what" — a rolling 30 days behaves very differently from a calendar month.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

Next

02-error-budgets — you have an SLI and an SLO target; the gap between that target and 100% is a budget you get to spend. You'll compute it, learn to spend it, and use it to make a real "ship vs. slow down" decision instead of arguing about one.