Prometheus Fundamentals

Why this matters

In track 07 module 06 you got cluster metrics by running one az aks enable-addons command — Azure deployed the agent, ran the pipeline, and stored the data for you. This module is where you do that yourself with Prometheus, the open-source metrics engine that the rest of the world (and, under the hood, much of Azure Monitor) runs on. Understanding Prometheus's pull model, how it discovers and scrapes targets, and how to write basic PromQL is the foundation for every metrics-based dashboard and alert you'll ever build — on any platform, not just Azure.

Concepts

The pull model and the /metrics endpoint

Prometheus does not receive metrics; it fetches them. Each thing you want to monitor exposes an HTTP endpoint (conventionally /metrics) that returns its current metrics in a simple text format — you saw the API server's own endpoint in module 00. Prometheus is configured with a list of targets and, on a fixed scrape interval (commonly 15s or 30s), makes an HTTP GET to each target's /metrics, parses the numbers, timestamps them with the scrape time, and stores them in its local time-series database (TSDB).

This is the opposite of the push-based Azure Monitor agents. The consequences are worth internalizing: Prometheus (the collector) controls the scrape rate, so a misbehaving app can't flood it; a failed scrape is itself a signal — Prometheus records up == 0 for a target it couldn't reach, which is often your first sign an app is down; and apps never need to know Prometheus's address, they just expose an endpoint and wait to be scraped. The cost is that Prometheus must discover what to scrape — which on Kubernetes it does automatically (below).

Exporters — metrics for things that don't speak Prometheus

Your own app can expose /metrics using a Prometheus client library. But the node's kernel, a Postgres database, or a Redis instance don't. An exporter is a small adapter process that reads some system's stats and re-exposes them in Prometheus format on a /metrics endpoint. The essential ones you'll meet immediately:

• node-exporter — runs as a DaemonSet (one per node) and exposes node-level CPU, memory, disk, and network metrics. This is the open-source equivalent of the node metrics Container Insights collected for you.
• kube-state-metrics — exposes the state of Kubernetes objects themselves as metrics: how many Deployments, how many Pods Ready, how many restarts, Pod phase. (Distinct from node-exporter: node-exporter is about the machine, kube-state-metrics is about the Kubernetes API's view.)
• cAdvisor (built into the kubelet) — per-container CPU/memory, the open-source counterpart to Container Insights' per-container view.

You rarely install these by hand — the Helm chart below bundles them.

kube-prometheus-stack: the batteries-included install

Installing Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics, and all the wiring by hand is tedious and error-prone. The community kube-prometheus-stack Helm chart (maintained under the Prometheus Community project) installs the whole lot preconfigured, using the Prometheus Operator — a controller that manages Prometheus via Kubernetes custom resources instead of hand-edited config files. This is the standard way to run Prometheus on Kubernetes and what you'll use for the entire track. One helm install gives you roughly what Container Insights gave you with one az command — except now you own the config, the retention, and the cost.

ServiceMonitor: how the Operator learns what to scrape

With the Prometheus Operator, you don't edit Prometheus's config file to add a scrape target. Instead you create a ServiceMonitor (or PodMonitor) custom resource that declaratively says "scrape any Service matching these labels, on this port, at this path." The Operator watches for ServiceMonitors and regenerates Prometheus's scrape config automatically. This is the same "declare desired state, a controller reconciles it" pattern you know from Deployments and from Gatekeeper Constraints in track 11 — applied to scrape configuration. A very common beginner failure (the diagnose-and-fix below) is a ServiceMonitor whose label selector or port name doesn't match the Service, so the target is silently never scraped.

PromQL, just enough to look around

PromQL is Prometheus's query language. This module keeps it minimal (module 02 goes deep). The three things to grasp now:

• An instant vector selector like up or up{job="node-exporter"} returns the current value of every matching series. The {label="value"} filter is how you narrow to specific series — and exact label matching is where "my query returns nothing" bugs come from.
• up is the single most useful starter query: it's 1 for every target Prometheus successfully scraped and 0 for every one it couldn't — an instant health map of your whole scrape config.
• Metrics have types: a counter only ever goes up (total requests ever) and is almost always wrapped in rate() before use (module 02); a gauge goes up and down (current memory, current temperature) and you read directly; a histogram buckets observations (request durations) for quantiles (module 02). Reading a raw counter's giant ever-increasing number is almost never what you want — recognizing counter vs. gauge now prevents confusion later.

Command reference

Flag-by-flag breakdown of the install command (exercise 1):

helm install kps prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

• kps — the release name; every resource gets it as a prefix, so you'll see kps-grafana, kps-kube-prometheus-stack-prometheus, etc.
• prometheus-community/kube-prometheus-stack — <repo>/<chart>.
• --namespace monitoring --create-namespace — install into (and create) a dedicated monitoring namespace, the convention for this stack.
• --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false — important for this track: by default the chart only scrapes ServiceMonitors carrying its own release label. Setting this to false tells Prometheus to consider every ServiceMonitor in the cluster, so the ones you write yourself in exercise 6 actually get picked up. (Without it, your custom ServiceMonitor is silently ignored — a classic first-time trap.)

Key PromQL forms (used in exercises, expanded in module 02):

• up — 1/0 health of every scrape target.
• up{job="node-exporter"} — narrow by the job label (exact match).
• up{job=~"node.*"} — =~ is a regex match; != and !~ are the negations.
• kube_pod_status_phase{phase="Running"} — a kube-state-metrics gauge.
• node_memory_MemAvailable_bytes — a node-exporter gauge, bytes of available RAM per node.

Hands-on exercises

All on your local kind cluster from track 03. Everything installs via Helm and costs nothing. Ensure helm is installed (track 03 used it).

1. (WSL2) Install kube-prometheus-stack.

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update
   helm install kps prometheus-community/kube-prometheus-stack \
     --namespace monitoring --create-namespace \
     --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false
   

   Wait a minute, then kubectl get pods -n monitoring. Expect Prometheus, Alertmanager, Grafana, node-exporter (a DaemonSet — one per node), and kube-state-metrics all reaching Running. You just reproduced, by hand, what Container Insights did with one Azure command.

2. (WSL2) See the exporters that got installed.

   kubectl get daemonset -n monitoring       # node-exporter, one pod per node
   kubectl get deploy -n monitoring          # kube-state-metrics, grafana, operator
   kubectl get servicemonitors -n monitoring # the pre-wired scrape declarations
   

   Note the stack shipped a dozen-plus ServiceMonitors already — it's scraping itself, the kubelet, the API server, etc., out of the box.

3. (WSL2) Open the Prometheus UI.

   kubectl port-forward -n monitoring svc/kps-kube-prometheus-stack-prometheus 9090:9090
   

   Browse to http://localhost:9090. Open Status → Targets: this is the live map of everything Prometheus is scraping and whether each is UP. This is the pull model made visible — every green target is one it successfully fetched /metrics from.

4. (WSL2) Run your first PromQL. In the Prometheus UI's Graph tab (or run these against the port-forward), evaluate:

   • up — every target's 1/0 health. Confirm the values match the Targets page.
   • node_memory_MemAvailable_bytes — available RAM per node (a gauge from node-exporter).
   • kube_pod_status_phase{phase="Running"} — one series per pod, 1 if in that phase (kube-state-metrics). Switch the filter to phase="Pending" and note most series read 0.

5. (WSL2) Deploy an app that exposes its own /metrics. Use a sample app with built-in Prometheus instrumentation:

   kubectl create namespace demo
   kubectl create deployment metrics-app -n demo --image=quay.io/brancz/prometheus-example-app:v0.5.0
   kubectl expose deployment metrics-app -n demo --port=8080 --name=metrics-app
   kubectl port-forward -n demo svc/metrics-app 8080:8080
   

   In another terminal: curl -s localhost:8080/metrics | head -30. You'll see http_requests_total{...} and friends — a real app pull target, exactly the format from module 00. Generate some traffic: for i in $(seq 20); do curl -s localhost:8080/ >/dev/null; done.

6. (WSL2) Tell Prometheus to scrape it with a ServiceMonitor. Prometheus isn't scraping metrics-app yet — you must declare it:

   kubectl apply -f - <<'EOF'
   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: metrics-app
     namespace: demo
   spec:
     selector:
       matchLabels:
         app: metrics-app
     endpoints:
       - port: "8080"
         path: /metrics
         interval: 15s
   EOF
   

   Wait ~30s, refresh Status → Targets in the Prometheus UI, and look for a demo/metrics-app target turning UP. Then query http_requests_total{namespace="demo"} — your app's own metric, now in Prometheus.

7. Diagnose and fix: a target that never scrapes because of a label mismatch. This is the single most common Prometheus-on-Kubernetes bug. Break it deliberately:

   kubectl apply -f - <<'EOF'
   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: metrics-app-broken
     namespace: demo
   spec:
     selector:
       matchLabels:
         app: wrong-name        # <-- does not match the Service's label
     endpoints:
       - port: "8080"
         path: /metrics
   EOF
   

   Refresh Status → Targets — metrics-app-broken produces no target at all (not a red one — no entry), because its selector.matchLabels matches no Service. Diagnose it the real way:

   kubectl get svc metrics-app -n demo --show-labels   # what labels does the Service actually have?
   

   You'll see the Service's label is app=metrics-app, not wrong-name. Fix by correcting the selector to app: metrics-app and re-applying; confirm the target now appears and goes UP. Lesson: a ServiceMonitor selector that matches nothing fails silently — no target, no error. Always verify against --show-labels.

8. (WSL2) Prove counter vs. gauge. Query http_requests_total{namespace= "demo"} (a counter) a few times over a minute while generating traffic — the raw number only ever climbs, and its absolute value is meaningless on its own. Then query node_memory_MemAvailable_bytes (a gauge) — it goes up and down. Write one sentence on why you'd wrap the counter in rate() (module 02) but read the gauge directly.

9. (WSL2) Optional AKS contrast. If you still have an AKS cluster from track 07, note that you could helm install this exact same stack there and get identical PromQL — the open-source path is cluster-portable in a way Container Insights (Azure-only) is not. Don't leave an AKS cluster running; az group delete when done.

10. (WSL2) Leave the stack installed — modules 02-07 all build on it. If you must reclaim resources: helm uninstall kps -n monitoring and kubectl delete namespace monitoring demo.

Independent challenge

No YAML given for the target — build it yourself using this module plus track 03 (Services, labels, selectors). Deploy a second instrumented workload of your choice into a new namespace — either another copy of the example app on a different port, or a real exporter like the Redis exporter in front of a Redis pod — and get Prometheus scraping it end to end by writing your own ServiceMonitor from scratch. Prove it worked three ways: the target shows UP on the Targets page, up{...} for it returns 1 in PromQL, and at least one of its application/exporter metrics returns data. Then deliberately reproduce the module's silent-failure mode once more — change one thing (a wrong port name, or a selector label that doesn't match) — confirm the target vanishes rather than erroring, and fix it by reading the Service's actual labels/ports. The skill being built is diagnosing "why isn't this being scraped?" without a single error message to guide you.

Common mistakes & troubleshooting

• Custom ServiceMonitor ignored entirely. By default the chart only picks up ServiceMonitors with its release label. Set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false (exercise 1) or add the release label to your ServiceMonitor. Symptom: your target never appears, with no error.
• Selector/port mismatch — the silent failure. A ServiceMonitor whose matchLabels or endpoints.port don't match the target Service produces no target at all — not a red/down one, just nothing. Always cross-check with kubectl get svc <name> --show-labels and its spec.ports.
• Confusing a down target with a missing target. up == 0 (red on the Targets page) means Prometheus found the target but the scrape failed (wrong port, app not listening) — a network/app problem. No entry at all means the ServiceMonitor didn't match anything — a config problem. Different fixes.
• Reading a raw counter as if it were meaningful. http_requests_total's absolute value is "requests since the process started" and resets to 0 on restart. You almost always want rate(...) (module 02), not the raw number.
• Expecting instant scrape. After creating a ServiceMonitor there's a short delay while the Operator regenerates config and the next scrape interval elapses — give it 30-60s before concluding it's broken.
• Putting high-cardinality labels on your own app's metrics. The module 00 warning applies to your instrumentation too: don't label http_requests_total with user_id or request_id. You control this in your app's client-library code.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

Next

02-promql-in-depth — now that metrics are flowing, learn to actually query them: rate/irate, aggregation across series, histograms and quantiles, and turning queries into recording and alerting rules.