Capstone Project: A Fully Observable System, End to End

Why this matters

Every module in this track added one capability in isolation — Prometheus scraping, PromQL, a Grafana dashboard, Loki logs, OpenTelemetry traces, Alertmanager, and finally correlation. Real observability is none of those alone; it's all three pillars flowing from one real application, wired together so that when something breaks you can move from that it broke to what broke to where it broke in minutes. This capstone is where you prove you can stand up that whole stack yourself — the vendor-neutral counterpart to the one-command Container Insights you enabled in track 07, except now you own every piece: the scrape config, the dashboards, the log pipeline, the tracing, the alert routing, and the correlation glue. There's no solution given and no quiz. Finishing this is what "you can make a system observable" means: you instrument it, you watch it, and you can debug it across all three pillars when it misbehaves.

The project

Take a two-service application (reuse the frontend→backend pair from module 05, an app from the Docker/Kubernetes tracks, or write a trivial pair where a request to the first triggers a call to the second) and make it fully observable on your kind cluster: scraped for metrics, dashboarded, logging to Loki, traced across both services with OpenTelemetry, alerting through Alertmanager, and correlatable by trace ID across all three pillars. Everything installs via Helm and costs nothing; you may optionally reproduce it on an AKS cluster from track 07 to contrast with Container Insights, but clean that up with az group delete when done.

The six required pillars, each drawing on a specific module:

  1. Prometheus scraping a real app (modules 01-02) — both services expose a /metrics endpoint, and a ServiceMonitor you wrote gets them scraped so their targets show UP and their own application metrics (request count, duration histogram) return data in PromQL. No metric may carry a high-cardinality label (user_id, request_id, trace_id).
  2. A Grafana dashboard with at least two panels on real PromQL (module 03) — built by hand on your app's metrics (not just an imported community board), driven by a $namespace (or $service) template variable, and including at minimum a rate panel and a quantile (p95/p99 latency) panel. At least one panel must use an aggregation with by.
  3. Loki collecting that app's logs (module 04) — both services emit structured (JSON) logs shipped to Loki by the agent, queryable in Grafana with LogQL, and — critically — each log line includes the request's trace_id in its content (never as a Loki label).
  4. OpenTelemetry tracing across at least two services (module 05) — a request through the frontend produces one joined trace whose backend span is correctly nested under the frontend span (proving trace context propagated), visible as a waterfall in Tempo (or Jaeger), routed through an OTel Collector.
  5. An Alertmanager rule that actually fires under a simulated condition (modules 02, 06) — a symptom-based alerting rule (e.g. error ratio or latency) with a severity label, a for: duration, and Alertmanager routing + grouping, which you force to fire by simulating the condition and watch travel Inactive → Pending → Firing → routed → delivered to a receiver you can observe.
  6. A written correlation note (module 07) for a specific incident scenario — use: "the checkout error-ratio alert just fired — walk me from the alert to the root-cause span." The note must give the ordered path (metric → logs → trace), the exact query at each step, which pillar answers that / what / where, and how the trace_id joins them.

Acceptance checklist

Work isn't done until you can demonstrate every one of these:

  • Both services expose /metrics; a ServiceMonitor you authored makes each target show UP on the Prometheus Status → Targets page (not missing, not up == 0).
  • Your app's own metrics return data in PromQL (a counter wrapped in rate() and a *_bucket histogram usable by histogram_quantile), and no metric carries a high-cardinality label — you can point at your metrics and justify every label as bounded.
  • A hand-built Grafana dashboard has ≥ 2 panels on your app's real PromQL, including a rate panel and a p95/p99 quantile panel (with sum by (le)), and a working $namespace/$service variable that re-targets every panel from a dropdown.
  • Both services log structured JSON to Loki; a LogQL query ({...} | json | level="error") returns their error lines, and each line contains the trace_id in its content (confirmed with | json | trace_id="<id>").
  • A request through the frontend produces one joined trace in Tempo/ Jaeger with the backend span nested under the frontend span; you can show that breaking propagation splits it into two traces and fixing it rejoins them.
  • An alerting rule (symptom-based, with severity + for:) is loaded by the Operator, and you forced it to fire — demonstrated the full transition and its delivery to a receiver via Alertmanager, with group_by set so a multi-pod incident is one notification.
  • You can take one trace_id and find the same string in all available pillars — the Tempo trace, the Loki logs of both services, and (if exemplars are wired) as a marker on the metric graph — proving the three pillars are genuinely joined, not just co-installed.
  • The written correlation note exists, is scenario-specific, lists the metric → log → trace steps in order with the exact query per step, and names which pillar tells you that / what / where.
  • At least one diagnose-and-fix from the track is reproducible on your system: you can deliberately break one link (a ServiceMonitor selector, a LogQL label, trace propagation, an alert-routing label, or a derived-field regex) and show it fails silently, then fix it.
  • If you built on AKS, all billable resources are cleaned up with az group delete.

Hints (not a solution)

  • Sequence it the way the track was ordered. Get metrics flowing and scraped first (you can't dashboard or alert on data you don't have), then build the dashboard, then add Loki logs, then tracing, then the alert, then write the correlation note last — by then you understand the system well enough to write a good one.
  • Reuse what you already installed. kube-prometheus-stack (modules 01-03), Loki + agent (module 04), Tempo + OTel Collector (module 05) are all still on your kind cluster if you didn't uninstall them. This capstone is mostly wiring your app into that existing stack, not reinstalling it.
  • Respect cardinality everywhere — it's what makes correlation work. The trace_id and user_id go in log content and span attributes, never in a metric label or a Loki index label (modules 00/04/05). The payoff is module 07's clickable metric → log → trace loop; you only get it if you kept the identifiers out of the indexes.
  • Test the bad path, not just the good one. For every pillar, the real proof is a silent failure fixed: a ServiceMonitor that matches nothing (no target), a LogQL selector that returns nothing, a trace that splits in two, an alert that fires into the void because its label matches no route, a derived-field regex that captures nothing. "It worked first try" usually means you didn't test the failure mode the track kept warning you about.
  • Force the alert; don't wait for it. Simulate the condition (drive 5xx traffic, or temporarily lower the threshold to what current traffic exceeds) so you watch it fire and route — an alert you've never seen fire is a guess (module 06's never-fires trap, and module 02's).
  • Make the correlation note real, not aspirational. Actually walk one reproducible error from alert to root-cause span, copy the real queries and the real trace_id you followed, and write down which pillar gave you which fact. That lived investigation is the deliverable — it's exactly what you'd paste into an incident review.
  • Don't gold-plate. Six pillars that genuinely connect beat a dozen half-wired dashboards. Depth on each acceptance item — especially the end-to-end trace-ID join — is the goal, not breadth.

Next

Before you move on: if any acceptance item is checked only because "it installed without error," go back and prove it the hard way — show the target actually UP, the panel actually drawing your app's PromQL, the trace actually joined across both services, the alert actually delivered to a receiver, and one trace_id actually found in all three pillars. A pillar you haven't seen hand off to the next one is a pillar you haven't verified. When every box is genuinely ticked — and you can start from a firing alert and reach the root-cause span by following one trace ID — you've finished the track: you can stand up metrics, logs, and traces on any platform, wire them together, and debug across all three, no managed monitoring add-on required.

This track built the visibility. Two later tracks build directly on it:

  • 13-service-mesh adds mesh-level observability on top of everything here — a service mesh (Istio/Linkerd) auto-generates the golden request metrics and distributed traces for service-to-service traffic without you instrumenting each app, and ships them straight into the same Prometheus/Grafana/Tempo you just stood up. The three-pillar foundation you built is exactly what the mesh plugs into.
  • 20-sre-practices formalizes SLOs and error budgets on top of these same metrics — the p95/p99 latency and error ratios you graphed and alerted on here become the SLIs behind formal service objectives, and the "alert on symptoms, not causes" discipline from module 06 becomes the basis of SLO-based alerting and error-budget policy.

You now have the foundation both of those build on.

Back to the track index · Back to main curriculum