Structured Logging and Log Aggregation with Loki

Why this matters

kubectl logs shows you one pod's logs, live, and loses them when the pod dies — the same limitation kubectl top had for metrics. In track 07 Container Insights solved this by shipping every container's logs into Log Analytics, queryable with KQL and durable across restarts. This module builds that same capability yourself with Loki, the open-source log aggregator that plugs into the same Grafana you already use for metrics. Along the way you'll learn the two things that make centralized logging actually work: structured logging (so logs are queryable, not just readable) and Loki's signature design choice — indexing labels, not full text — which is why it stays cheap where an ELK stack gets expensive.

Concepts

Structured logging — logs a machine can query

A log line like Error processing order 4711 for user bob: timeout is readable but hard to query — extracting "all errors for user bob" means fragile text parsing. Structured logging emits logs as key-value data (usually JSON): {"level":"error","msg":"processing failed","order_id":4711, "user":"bob","reason":"timeout"}. Now a log aggregator can filter on level="error" and extract user reliably. This is the log-pillar counterpart to module 00's cardinality lesson: put the rich, high-cardinality detail (order_id, user, request_id) in the structured log fields — exactly where it doesn't belong in a metric. Structuring your app's logs is the single highest -leverage thing you can do to make them useful later, and it's a code change in your app, not a tooling change.

Loki's big idea — index labels, not the log text

Traditional log systems (Elasticsearch/ELK) build a full-text index over every word in every log line — powerful (search any word instantly) but expensive in storage and compute, and the reason ELK clusters are heavy to run. Loki makes the opposite trade: it indexes only a small set of labels (like Prometheus — namespace, pod, app, level) and stores the raw log content unindexed in cheap object storage. You query by label to narrow to a stream, then Loki brute-force greps the content of just that narrowed set. The result: dramatically cheaper storage and operation, at the cost of slower arbitrary-word search across everything. Loki is often described as "Prometheus, but for logs" — deliberately, because it reuses the same label model, so {namespace="demo", app="metrics-app"} selects a log stream the same way a metric selector selects series.

The cardinality trap comes back — this time in log labels

Because Loki indexes labels, the same cardinality discipline from module 00 applies to Loki labels. Putting a high-cardinality value (request_id, user_id, a full URL) into a Loki label creates a separate stream per value and blows up Loki's index — the exact mistake, in a new place. The rule: labels are for the low-cardinality dimensions you slice by (namespace, app, level, pod); the high-cardinality detail stays in the log line content and is extracted at query time with a LogQL filter/parser, not indexed. If you internalized "don't put user_id in a metric label," this is the same sentence with "Loki" swapped in.

The agent — Promtail / Grafana Alloy ships the logs

Loki stores and queries logs but doesn't collect them. A collection agent runs on each node (a DaemonSet, like node-exporter), tails every pod's log files, attaches Kubernetes labels (namespace, pod, container), and pushes the streams to Loki. Historically this was Promtail; the current recommended agent is Grafana Alloy (Promtail is being deprecated in favor of it), but they play the same role. This is the open-source equivalent of the Container Insights agent DaemonSet you saw ship logs to Log Analytics — except you can see and configure exactly what it collects, relabels, and drops (dropping noisy logs before ingestion is a real cost lever, just like trimming verbosity was in track 07).

LogQL — Loki's query language

LogQL deliberately mirrors PromQL. A query has two parts: a stream selector in {} (label matchers, mandatory) and an optional line filter / pipeline after it:

• {namespace="demo"} — every log line from that namespace (stream selector only).
• {namespace="demo"} |= "error" — lines containing the substring "error" (|= contains, != doesn't contain, |~ regex).
• {namespace="demo"} | json | level="error" — parse each line as JSON, then filter on the extracted level field (this is where structured logging pays off).
• sum(rate({namespace="demo"} |= "error" [5m])) — LogQL can even compute metrics from logs: the per-second rate of error lines. This is the bridge that lets you graph log-derived rates next to real metrics, and it previews module 07's correlation.

Command reference

Flag-by-flag breakdown of the install (exercise 1):

helm install loki grafana/loki-stack --namespace monitoring --set promtail.enabled=true --set grafana.enabled=false --set loki.isDefault=false

• grafana/loki-stack — the bundle chart: Loki + Promtail agent (+ optionally Grafana, which you already have).
• --namespace monitoring — install alongside the Prometheus stack so one Grafana can reach both.
• --set promtail.enabled=true — deploy the log-shipping agent DaemonSet.
• --set grafana.enabled=false — don't install a second Grafana; you have one from kube-prometheus-stack.
• --set loki.isDefault=false — keep Prometheus as Grafana's default data source; Loki is added but not the default.

LogQL pipeline {namespace="demo"} | json | level="error" | line_format "{{.msg}}"

• {namespace="demo"} — stream selector: the indexed labels, narrows the set.
• | json — parse each line as JSON, exposing its fields (level, msg, user) as queryable labels for this query only (not indexed).
• | level="error" — filter on an extracted field — only possible because the logs are structured.
• | line_format "{{.msg}}" — reformat the displayed line to just the msg field. Optional prettifying.

Hands-on exercises

Continue on the kind cluster with kube-prometheus-stack and Grafana from modules 01-03.

1. (WSL2) Install Loki and its agent.

   helm repo add grafana https://grafana.github.io/helm-charts
   helm repo update
   helm install loki grafana/loki-stack \
     --namespace monitoring \
     --set promtail.enabled=true --set grafana.enabled=false --set loki.isDefault=false
   kubectl get pods -n monitoring | grep -E 'loki|promtail'
   

   Expect a loki-0 pod and a promtail DaemonSet (one per node) reaching Running. This is your open-source log pipeline: Promtail tails pod logs → Loki stores them.

2. (WSL2) Add Loki as a Grafana data source. In Grafana (port-forward svc/kps-grafana 3000:80 from module 03) go to Connections → Data sources → Add data source → Loki, URL http://loki:3100, Save & test. (Depending on chart versions it may already be present.)

3. (WSL2) Deploy a structured-logging app and view its logs. Deploy something that logs JSON:

   kubectl create namespace demo 2>/dev/null; true
   kubectl create deployment jsonlog -n demo --image=busybox -- /bin/sh -c \
     'i=0; while true; do lvl=info; [ $((i % 5)) -eq 0 ] && lvl=error; echo "{\"level\":\"$lvl\",\"msg\":\"processed\",\"order_id\":$i,\"user\":\"user-$((RANDOM%50))\"}"; i=$((i+1)); sleep 1; done'
   

   Wait ~30s, then in Grafana Explore pick the Loki source and run {namespace="demo"}. You should see the JSON log lines flowing — durable and centralized, unlike kubectl logs.

4. (WSL2) Filter with a line filter. Run {namespace="demo"} |= "error" — only the error lines. Note this is a substring grep over the narrowed stream (Loki's brute-force-after-label- narrowing model in action).

5. (WSL2) Parse structured fields — the payoff of JSON logs. Run {namespace="demo"} | json | level="error". Loki parses each line as JSON and filters on the extracted level field — reliable, not fragile text matching. Then extract a high-cardinality field the right way: {namespace="demo"} | json | user="user-7" — user was in the log content (queryable), never a Loki label (which would've exploded the index). This is module 00's cardinality rule, correctly applied.

6. (WSL2) Compute a metric from logs. Run sum(rate({namespace="demo"} |= "error" [5m])) in Explore. Loki returns a rate — a metric derived from log lines, graphable next to your Prometheus panels. Note how this bridges the logs and metrics pillars (module 07 leans on this).

7. Diagnose and fix: logs missing because of a label mismatch. Run a query with a wrong label value and get nothing: {namespace="demo", app="wrong-name"} → no results. The bug: the stream's real labels don't include app="wrong-name". Diagnose the real way — click the label browser in Explore (or run {namespace="demo"} and inspect the labels on a returned line) to see the actual labels Promtail attached (namespace, pod, container, app/job depending on config). Fix the query to use a label that exists. Lesson (identical in spirit to module 01's ServiceMonitor selector and module 03's "No data"): a stream selector that matches no labels returns nothing silently — always confirm labels against real streams.

8. (WSL2) See the labels-vs-content tradeoff concretely. Run {namespace="demo"} | json | order_id="100". This works but Loki had to scan the content of the whole demo stream to find it (no index on order_id) — fine here, slow at scale. Contrast: {namespace="demo"} narrows by indexed label instantly. Write one sentence on why order_id must stay content, not become a label.

9. (WSL2) Optional — the ELK alternative. You won't install it, but note the contrast: an ELK/Elasticsearch stack would full-text-index every field of these JSON logs, making arbitrary-word search fast but costing far more storage/compute. Loki chose cheap+label-scoped over fast+everywhere. In a sentence, note which you'd pick for a high-volume, cost-sensitive cluster (Loki) vs. a security-forensics use case needing arbitrary historical search (ELK's full-text index earns its cost).

10. (WSL2) Leave Loki installed for module 07's correlation work. To reclaim resources: helm uninstall loki -n monitoring.

Independent challenge

No queries given — build them yourself using this module plus module 00 (which data is high vs. low cardinality) and module 03 (Explore, panels). Take a workload that emits structured logs (the jsonlog app, or better, make a small app of your own log JSON with level, a low-cardinality component field, and a high-cardinality request_id). First, write LogQL that answers three questions an on-call engineer asks: (1) show only error-level lines for one component, (2) count the error rate per component over 5 minutes as a graphable metric, and (3) find every log line for one specific request_id. Crucially, for each field you use, state whether it should be a Loki label or stay in the log content, and justify it with the cardinality rule. Then add a Grafana panel that puts the log-derived error rate (question 2) next to a real Prometheus metric panel for the same app — proving logs and metrics can share one dashboard. The skill is knowing what to index vs. what to extract at query time, which is the entire art of using Loki well.

Common mistakes & troubleshooting

• Unstructured logs you can't query. Free-text logs force fragile substring matching. Emit JSON/logfmt so | json/| logfmt can extract fields — this is an app code change, do it early.
• High-cardinality Loki labels. request_id/user_id/full-URL as a Loki label explodes its index — the module-00 cardinality trap, relocated. Keep them in the log content and extract with LogQL at query time.
• Empty stream selector results. A {} selector matching no real labels returns nothing silently. Use Explore's label browser to see the actual labels Promtail/Alloy attached before guessing.
• Expecting instant search over everything. Loki isn't full-text-indexed; broad, label-less content searches over huge time ranges are slow by design. Always narrow by label first, then filter content.
• Installing a second Grafana. loki-stack bundles Grafana; set grafana.enabled=false so you keep one Grafana wired to both Prometheus and Loki, not two disconnected ones.
• Not dropping noisy logs. Every ingested line costs storage — the same verbosity/cost lesson as track 07's Log Analytics. Drop or relabel noisy streams at the agent (Promtail/Alloy) before they hit Loki.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

Next

05-distributed-tracing-and-opentelemetry — you now have two of three pillars (metrics + logs) flowing into one Grafana. Add the third and least-familiar one: distributed traces with OpenTelemetry, showing how a single request moves across services.