Correlating the Three Pillars

Why this matters

This is the module the whole track was building toward. Collecting metrics, logs, and traces separately is table stakes; the payoff is navigating between them during an incident — a metric alert fires, you jump to the exact logs behind the spike, and from a log line you jump to the specific trace that shows which service was at fault. Three tools become one debugging flow. The connective tissue is a shared identifier (the trace ID) threaded through all three pillars, plus Grafana features (exemplars, data links, derived fields) that turn "correlation in principle" into a literal click. Teams that collect all three pillars but never wire them together do ten times the work during an incident; this module is how you avoid that.

Concepts

The trace ID is the join key

Everything here rests on one idea: the same trace_id appears in all three pillars for a given request. You set this up deliberately in module 05's challenge — each service logs the trace_id of the request it's handling, and that ID is of course the identity of the trace itself. So a trace_id found in a log line is a direct pointer to the trace; a trace_id attached to a metric sample (an exemplar, below) points to a representative trace behind that data point. The trace ID is the primary key that joins the three otherwise-separate datastores — which is why module 00 insisted it live in log content and trace attributes (queryable, join-able) and never in an index label (it's per-request, unbounded). Correlation is the reward for having respected cardinality all along.

Exemplars — a trace ID attached to a metric

A metric is aggregate — histogram_quantile tells you p99 latency is 2s but not which request was slow. An exemplar fixes exactly this: it's a trace_id (plus timestamp) attached to a specific histogram observation, so a single data point on your latency graph carries a pointer to one real trace that contributed to it. In Grafana, exemplars render as little diamonds on the graph; click one and you jump straight to that trace in Tempo. This is the metric → trace bridge — the direct answer to "the p99 spiked, show me a slow request." Exemplars require the app's instrumentation to emit them and Prometheus to be configured to store them, but conceptually they're just "the metric point remembers one example request."

Grafana's glue: data links and derived fields

Grafana makes the jumps clickable with two features:

• Derived fields (Loki data source config) — a regex that extracts a trace_id from log lines and turns it into a link to the Tempo data source. Result: every log line in Grafana showing a trace_id becomes a one-click jump to that trace. This is the log → trace bridge.
• Data links / trace-to-logs (Tempo data source config) — the reverse: from a span in a Tempo trace, a link back to Loki filtered by the trace's ID (and time window), so you go trace → logs to read what that service logged while handling the request.

Wired together, these two closures plus exemplars give you a full loop: metric → (exemplar) → trace → (trace-to-logs) → logs, and log → (derived field) → trace. That loop is observability paying off.

The incident workflow, concretely

The canonical flow this all enables, in order:

1. Metric alert fires (module 06): "checkout error ratio > 5%." You know that something's wrong and roughly where (the checkout service) and when.
2. Pivot to logs (module 04): in Grafana, filter Loki to the checkout service and the alert's time window, read the actual error messages, and grab a trace_id from a failing line.
3. Pivot to the trace (module 05): click the trace ID, open the waterfall, and see which downstream hop (payments? the DB?) actually failed or was slow — the root cause.

Each pillar hands to the next: metric = that it's happening, log = what the error is, trace = where in the call graph. Doing this by copy-pasting IDs between three separate tools works but is slow; wiring the links (above) makes it clicks. Either way, the thinking — narrow with metrics, read with logs, locate with traces — is the durable skill.

Why "one pane of glass" matters here

You deliberately used Grafana for all three pillars (Prometheus + Loki + Tempo as three data sources) instead of three separate UIs. This module is why: correlation across pillars is dramatically easier when they share one tool that can link between them and show them side by side (a split view with the metric graph, the logs, and the trace all open). Container Insights gave you metrics+logs in one Azure portal but weak tracing; this open-source stack gives you all three, correlatable, in one Grafana — the capability the whole track assembled.

Command reference

Mostly Grafana data-source configuration (where the correlation glue lives) and the queries that walk the pillars.

Derived-field config (Loki data source → Derived fields):

• Name: trace_id — the field label shown on the log line.
• Regex: "trace_id":"(\w+)" — captures the ID out of your JSON log line (adjust to your log format). The capture group is the extracted value.
• Query: ${__value.raw} — feeds the captured ID into…
• Data source: Tempo — …a link that opens that trace ID in Tempo. Result: a clickable trace link on every log line.

Trace-to-logs config (Tempo data source → Trace to logs):

• Data source: Loki — where to send you.
• Tags / filter: map the span's service to a Loki label and use the trace's time range, so the link lands on the right logs, narrowed to the request's window.

Hands-on exercises

This module needs all three pillars from earlier modules installed on your kind cluster: kube-prometheus-stack (metrics, modules 01-03), Loki (logs, module 04), and Tempo + OTel Collector (traces, module 05), plus ideally the two-service, trace-ID-logging app from module 05's challenge. If you uninstalled any, reinstall before starting.

1. (WSL2) Confirm all three data sources in one Grafana. In Grafana Connections → Data sources, confirm Prometheus, Loki, and Tempo all exist and test green. This one-Grafana-three-sources setup is the precondition for everything below.

2. (WSL2) Do the correlation manually first (the thinking, before the glue). Drive traffic (including some errors) through your instrumented app. Then, by hand: (a) in a Prometheus/Grafana panel, spot an error-rate or latency bump; (b) switch to Loki in Explore, query {namespace="demo"} | json | level="error" for that window, and copy a trace_id from a failing line; (c) switch to Tempo, paste the trace_id, and open the waterfall to find the slow/failed span. You just walked metric → log → trace by copy-paste. Feel how the trace ID is the join key.

3. (WSL2) Wire the log → trace link (derived field). In the Loki data source settings, add a Derived field named trace_id with a regex that captures your log's trace ID (e.g. "trace_id":"(\w+)"), query ${__value.raw}, data source Tempo. Save. Re-run the Loki query from exercise 2 — now each log line shows a trace_id with a clickable button that jumps straight to that trace in Tempo. The copy-paste from exercise 2 is now one click.

4. (WSL2) Wire the trace → logs link. In the Tempo data source settings, enable Trace to logs, point it at Loki, and configure it to filter by the trace's time range (and, if you set service labels, the service). Open a trace, click a span, and use the "Logs for this span" link to land in Loki showing what that service logged during the request. You now have the loop closed in both directions.

5. (WSL2) Use the Explore split view. In Explore, click Split: put Loki on the left and Tempo on the right. Click a trace link from a log line and watch the trace open beside the logs — metric spike context, the log detail, and the trace, visible together. This side-by-side is the "one pane of glass" payoff in action.

6. (WSL2) Exemplars: metric → trace (if your app emits them). If your instrumentation emits exemplars and Prometheus is configured to store them, enable exemplars on the Prometheus data source and graph your latency histogram query — look for diamond markers on the graph. Click one to jump to a representative trace behind that data point. (If your app doesn't emit exemplars, note conceptually that this is the metric → trace bridge and move on — it requires app-side support.)

7. Diagnose and fix: correlation breaks because the trace ID isn't in the logs. The whole loop depends on the trace_id being present and consistently formatted in log lines. Reproduce a break: use a service that logs errors but doesn't include the trace_id field (or logs it under a different key like traceID your derived-field regex doesn't match). Try to pivot from its logs to a trace — there's no link, or the regex captures nothing. Diagnose: inspect a raw log line and compare its actual field name/ format against your derived-field regex. Fix either the app (log the trace_id in the expected field) or the regex (match the field the app actually uses). Lesson: correlation is only as good as the weakest pillar's instrumentation — a missing or mis-keyed trace ID silently breaks the loop, the same silent-mismatch family as every other diagnose-and-fix in this track (scrape selector, LogQL selector, trace propagation, alert routing).

8. (WSL2) Write the incident narrative. Pick one real error you can reproduce and write out the full path you took: the metric that flagged it, the log query and the error text you found, the trace ID you followed, and the exact span that was the root cause. This written "how I'd debug this" narrative is the capstone deliverable in miniature — practice it here.

Independent challenge

No steps given — run the whole loop yourself using every prior module. Starting from only an alert (reuse module 06's firing critical alert on the demo app), reconstruct a complete incident investigation end to end without looking back at exercise 2's steps: from the alert, narrow the time and service with metrics (modules 02-03); pivot to Loki logs to read the actual failure and extract a trace_id (module 04); follow that ID into Tempo to identify the exact failing/slow span across your two services (module 05); and write a short root-cause note naming which pillar told you what. Then improve the setup: identify the one place your correlation is still manual (a missing derived field, a service not logging trace_id, no trace-to-logs link, or no exemplars) and wire it so the same investigation next time is one click shorter. The deliverable is both the investigation and the one correlation improvement — because in real incident response, the follow-up "make this faster next time" is as valuable as the fix itself (a thread track 20 picks up as SLOs and error budgets).

Common mistakes & troubleshooting

• No shared trace ID across pillars. If the app doesn't log its trace_id (or logs it under a key your derived-field regex doesn't match), the log→trace jump is impossible. The join key must be present and consistently formatted in every pillar.
• trace_id promoted to an index label to "make it searchable." Don't — it's unbounded (module 00) and explodes the index. Keep it in log content/span attributes and extract at query time; that's already searchable.
• Three separate UIs. Correlating across three disconnected tools is slow. Put Prometheus, Loki, and Tempo in one Grafana so the links and split view work.
• Expecting a trace for every logged request. Sampling (module 05) means the trace behind a given log line may not have been retained — a normal gap, not a broken link. Keep 100% of errors via tail sampling if this hurts.
• Derived-field regex mismatch. The most common reason the log→trace button doesn't appear: the regex doesn't match the log's actual trace_id field. Test it against a real raw line.
• Correlating by timestamp alone. Lining up three tools by clock is fragile (clock skew, busy windows). The trace ID is an exact join; use it, and use time only to narrow.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

Next

08-capstone-project — assemble everything: a scraped app, a Grafana dashboard, Loki logs, cross-service tracing, an alert that actually fires, and a written plan for correlating all three pillars during a real incident.