Grafana: Dashboards and Visualization

Why this matters

PromQL in the Prometheus UI is fine for one-off queries, but nobody watches a query box during an incident — they watch a dashboard. In track 07 the Container Insights workbooks were built and styled for you; here you build the dashboards yourself in Grafana, the open-source visualization layer that sits on top of Prometheus (and, later, Loki and Tempo — one pane of glass for all three pillars). Learning to build a good dashboard — the right panels on real PromQL, variables that make one dashboard serve many services, and knowing when to just import a community board — is the difference between a wall of pretty graphs and something that actually answers questions fast.

Concepts

Data sources — Grafana queries, it doesn't store

Grafana stores no metrics of its own. It's a query-and-visualize front end that connects to data sources: Prometheus for metrics, Loki for logs (module 04), Tempo for traces (module 05). The kube-prometheus-stack you installed already wired Grafana to its Prometheus as the default data source, so you can build panels immediately. The key mental model: a Grafana panel is a saved PromQL (or LogQL, or TraceQL) query plus a choice of how to draw it. Everything you learned in module 02 is what goes in the query box; this module is about the drawing and the organizing.

Panels and visualization types — match the viz to the question

A panel is one visualization of one or more queries. The type you pick should match the shape of the question:

• Time series (line graph) — the default for anything trending over time: request rate, latency, CPU. 90% of panels.
• Stat / gauge — a single current number (or number vs. threshold): "error rate right now", "pods ready". Good for at-a-glance status tiles.
• Bar gauge / table — comparing a value across a label dimension (top-10 slowest endpoints, per-namespace request counts).
• Heatmap — histogram buckets over time, the natural way to see a latency distribution shift (the visual counterpart to histogram_quantile).

Choosing a stat panel for a trend, or a giant table where a graph would do, is the most common "technically correct but useless" dashboard mistake. Match the viz to whether the question is "what's the trend", "what's the value now", or "how do parts compare".

Variables and templating — one dashboard, many targets

A dashboard hard-coded to namespace="demo" is useless for your other 20 namespaces. Template variables fix this: you define a variable like $namespace whose values are populated by a query (label_values(kube_pod_info, namespace)), Grafana renders a dropdown at the top of the dashboard, and your panels use namespace="$namespace" instead of a literal. Now one dashboard serves every namespace, switchable from a dropdown. This is the single biggest force-multiplier in Grafana — the same "parameterize the reusable thing" idea as a Gatekeeper ConstraintTemplate's parameters (track 11) or a Terraform variable (track 09), applied to dashboards. Variables can also be multi-value and All, and chain (a $pod variable filtered by the chosen $namespace).

Importing community dashboards — don't build what exists

For common systems (Kubernetes cluster health, node-exporter, NGINX, Postgres) excellent dashboards already exist on grafana.com/dashboards, each with a numeric ID you paste into Grafana's import screen. kube-prometheus-stack even ships a set pre-loaded. The discipline: import a community dashboard for standard/off-the-shelf systems, build your own only for your app's custom metrics. Reinventing the Kubernetes-cluster dashboard by hand is wasted effort; a dashboard for your business metrics (orders_total, checkout_duration) is something only you can build. Imported dashboards also teach you PromQL — open their panels and read the queries.

Dashboards as code (a forward glance to GitOps)

Clicking a dashboard together in the UI is how you learn, but a click-built dashboard lives only in Grafana's database and vanishes if the pod is recreated. Grafana dashboards are ultimately JSON, which means they can be version-controlled and provisioned declaratively — exactly the config-as-code discipline from track 10. kube-prometheus-stack provisions its dashboards from ConfigMaps this way. You won't do a full GitOps dashboard pipeline here, but internalize now that the real home for a dashboard you care about is a JSON file in Git, not a hand-clicked artifact in one Grafana instance — a click-built board is a draft, not a deliverable.

Command reference

Grafana is mostly a web UI, so this is a mix of access commands and UI-navigation "where do I click" reference.

Flag-by-flag breakdown of the two commands worth dissecting:

kubectl get secret -n monitoring kps-grafana -o jsonpath='{.data.admin-password}' | base64 -d

• kps-grafana — the Secret the chart created for Grafana's admin creds.
• -o jsonpath='{.data.admin-password}' — extract just the password field (Secret data is base64).
• | base64 -d — decode it. The default username is admin.

Variable query label_values(kube_pod_info, namespace)

• label_values(<metric>, <label>) — a Grafana-specific templating function (not PromQL) that returns the distinct values of <label> across series of <metric>. Here: every namespace that has pods — the dropdown's options.
• Use label_values(kube_pod_info{namespace="$namespace"}, pod) for a chained $pod variable that only lists pods in the selected namespace.

Hands-on exercises

Continue on the kind cluster with kube-prometheus-stack, the demo app, and the recording rule from module 02. Generate some traffic first so panels have data (for i in $(seq 500); do curl -s localhost:8080/ >/dev/null; done against the port-forwarded metrics-app).

1. (WSL2) Log into Grafana.

   kubectl get secret -n monitoring kps-grafana -o jsonpath='{.data.admin-password}' | base64 -d; echo
   kubectl port-forward -n monitoring svc/kps-grafana 3000:80
   

   Browse to http://localhost:3000, log in as admin with that password.

2. (WSL2) Confirm the data source. Go to Connections → Data sources and confirm a Prometheus data source exists and its Test button passes. This is the pre-wired connection; note Grafana stores nothing itself — it's querying the Prometheus you installed in module 01.

3. (WSL2) Build your first panel on real PromQL. Dashboards → New → New dashboard → Add visualization, pick the Prometheus data source, and in the query box enter sum by (code) (rate(http_requests_total{namespace="demo"}[5m])). Set the visualization to Time series. You should see request rate split by status code — the module 02 query, now drawn. Give the panel a title ("Demo app request rate by status").

4. (WSL2) Add a stat panel and a latency panel. Add two more panels to the same dashboard:

   • A Stat panel showing current total rate: sum(rate(http_requests_total{namespace="demo"}[5m])).
   • A Time series panel for p95 latency: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{namespace="demo"}[5m]))). Save the dashboard ("Demo app"). You now have a multi-panel board on your own app's real metrics — a capstone requirement.

5. (WSL2) Add a $namespace variable and templatize. Dashboard settings → Variables → New variable: name namespace, type Query, data source Prometheus, query label_values(kube_pod_info, namespace). Save. A dropdown appears at the top. Now edit each panel's query, replacing namespace="demo" with namespace="$namespace". Switch the dropdown to monitoring and watch the panels re-target a different namespace with no query rewriting. This is the reusability payoff.

6. (WSL2) Import a community dashboard. Dashboards → New → Import, enter ID 1860 (the popular "Node Exporter Full" board), select your Prometheus data source, and import. Explore it — CPU, memory, disk, network per node, all pre-built. Open one panel's edit view and read its PromQL: you're learning query patterns from a board you didn't have to write. Note the discipline: you'd never hand-build this; you would hand-build panel 3/4's app-specific board.

7. Diagnose and fix: a panel showing "No data" because of a label that doesn't exist. Add a panel with a subtly wrong query: sum by (code) (rate(http_requests_total{namespace="demo",status="200"}[5m])). It shows No data. The bug: this app labels status as code, not status, so status="200" matches no series. Diagnose it the real way — in the Prometheus UI (or Grafana's Explore), run http_requests_total{namespace="demo"} and read the actual label names on the returned series. Fix the query to use code="200". Lesson (straight from module 02's "binary op returns nothing"): "No data" is almost always a label name/value mismatch, not a broken data source — verify labels against real series, don't guess.

8. (WSL2) Use Explore to prototype before building panels. Open Explore (compass icon), pick Prometheus, and iterate on a query live before committing it to a panel. This is the fastest debug loop — build the query in Explore, confirm it returns data, then paste it into a panel. Adopt this habit; it prevents the exercise-7 confusion.

9. (WSL2) Export your dashboard as JSON. Dashboard settings → JSON Model (or the share/export menu). Copy the JSON. This is what "dashboards as code" means — this file could live in Git and be provisioned automatically (track 10's config-as-code idea). Note it's a plain, version-controllable artifact, not locked inside Grafana.

Independent challenge

No panel specs given — design the board yourself using this module plus module 02 (the queries) and module 01 (which metrics exist). Build a single, reusable "service health" dashboard for the demo app (or any workload you scrape) with at least four panels that together answer the questions an on-call engineer actually asks: is it up, how much traffic, how many errors, how slow. Requirements: at least one panel must use a rate, one must use a ratio (error fraction), one must show a quantile (p95/p99 latency), and the whole dashboard must be driven by a $namespace template variable so switching the dropdown re-targets every panel. Then find and import one relevant community dashboard for a system you didn't write (node-exporter, kube-state, or the cluster), and in a sentence justify why you built the first board by hand but imported the second. The skill is composing a dashboard that answers questions in the order an incident unfolds, not just displaying whatever metrics happen to exist.

Common mistakes & troubleshooting

• "No data" panic. Almost always a label name/value mismatch or a time range with no samples — not a broken Grafana. Verify the exact query returns data in Explore/the Prometheus UI and read the real label names on the series before assuming anything is broken.
• Wrong visualization for the question. A trend in a Stat panel, or a giant table where a graph belongs. Match the viz to "trend vs. current value vs. comparison across a dimension".
• Hard-coded namespaces/labels. A dashboard pinned to one namespace serves one namespace. Templatize with variables from the start — retrofitting later means editing every panel.
• Rebuilding what exists. Hand-building a Kubernetes/node-exporter dashboard that's a free community import is wasted effort. Import standard boards; hand-build only your app's custom metrics.
• Treating a click-built dashboard as durable. It lives only in Grafana's DB and can be lost. Export the JSON and keep the ones you care about in Git (dashboards as code).
• Ignoring the data source when importing. Community dashboards prompt for which Prometheus data source to bind to — pick yours, or every panel shows No data despite the queries being correct.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

Next

04-logging-and-log-aggregation-loki — you've built out the metrics pillar end to end (collect → query → visualize). Now add the second pillar: centralized logs with Loki, viewable in this same Grafana, and the labels-vs-full-text tradeoff that makes Loki cheap.