Distributed Tracing and OpenTelemetry

Why this matters

This is the pillar the Azure-native tracks barely touched. Metrics told you request latency spiked; logs told you what one service logged — but when a single checkout request fans out across an API gateway, an orders service, and a database, neither tells you where the time actually went. Distributed tracing does: it follows one request across every service and shows you the exact hop that was slow or failed. OpenTelemetry (OTel) is the vendor-neutral standard for producing that data — the CNCF project that replaced a pile of competing formats — and it's the least-familiar and often most-revealing pillar for anyone running microservices. This module gets a real trace flowing across two services and into Grafana.

Concepts

Spans and traces — the anatomy

A span is a single timed operation: "handle HTTP GET /checkout", "query the orders DB", "call the payments service". Each span has a name, a start and end time (so a duration), and key-value attributes (http.method, db.statement, http.status_code). A trace is a tree of spans representing one request's full journey: a root span (the initial request) with child spans for each downstream operation it triggered, nested by parent-child relationships. Rendered on a timeline (a "waterfall"), a trace shows you visually that the request spent 5ms in the gateway, then 800ms blocked on the database — the "where did the time go" answer metrics and logs can't give. This is the high-cardinality pillar from module 00 made concrete: each trace is tied to a specific trace_id, exactly the kind of per-request identifier you keep out of metric labels.

Trace context propagation — the thing that actually makes it work

The magic and the fragility of tracing both live here. For spans from different services to join into one trace, each service must pass a trace context — a trace_id (shared by every span in the trace) and the current span_id (the parent for the next hop) — to the next service, almost always as an HTTP header. The standard is W3C Trace Context, a header named traceparent. When service A calls service B, A injects traceparent into the outgoing request; B extracts it and makes its spans children of A's. If any service fails to propagate that header, the trace breaks into two disconnected traces — B starts a brand-new root instead of continuing A's. This is the single most common tracing bug and the diagnose-and-fix below: a missing span because context wasn't propagated. Auto-instrumentation usually handles propagation for you; manual or mixed setups are where it breaks.

Instrumentation — auto, manual, and where spans come from

Spans have to be created somewhere. Two ways:

Auto-instrumentation — an OTel library/agent for your language hooks into common frameworks (HTTP servers, clients, DB drivers) and emits spans and propagates context automatically, with little or no code change. This is how you get value fast.
Manual instrumentation — you call the OTel SDK in your own code to start and end spans around business logic the auto-instrumentation can't see ("scored the fraud model"), and add custom attributes.

Real systems use both: auto for the plumbing (HTTP/DB), manual for the domain-specific spans that make a trace actually explain your business logic.

The OpenTelemetry Collector — the pipeline in the middle

Apps could send traces straight to a tracing backend, but the standard architecture puts an OpenTelemetry Collector in between: a vendor-neutral service that receives telemetry (via OTLP, the OTel protocol), processes it (batching, sampling, dropping, adding attributes), and exports it to one or more backends (Tempo, Jaeger, or a cloud vendor). Its config is three pipelines of receivers → processors → exporters. Why bother? It decouples your apps from the backend (switch Tempo→Jaeger by editing the Collector, not every app), centralizes sampling (below) and cost control, and can process all three signals — traces, metrics, and logs — through one component. It's the same "put a configurable pipeline between producers and storage" role that Promtail/Alloy played for logs, generalized.

Sampling — you can't afford to keep every trace

Traces are the heaviest pillar (module 00): one per request, each with many spans. At scale you can't store them all, so you sample — keep a fraction. Head sampling decides at the start of the trace (e.g. keep 10% at random) — cheap but might drop the interesting slow/errored traces. Tail sampling decides after the trace completes, so you can keep 100% of errors and slow traces and 1% of the boring fast ones — smarter, done in the Collector, costlier to run. The tradeoff to remember: sampling is why a specific request you're hunting might have no trace — a normal, expected consequence, not a bug, and a real gotcha when correlating in module 07.

Tempo vs. Jaeger — the backend

Something has to store and let you view traces. Grafana Tempo is the Loki-of-traces: cheap object-storage backend, and it renders traces inside the same Grafana you use for metrics and logs (ideal for this track's one-pane goal). Jaeger is the older, standalone, CNCF tracing UI — excellent and common, but a separate UI. This module uses Tempo as primary (to keep one Grafana) and notes Jaeger as the well-known alternative, the same "pick one, know the other exists" split as Loki-vs-ELK in module 04.

Command reference

Command / config key	What it does	Example
`helm install tempo grafana/tempo`	Installs the Tempo tracing backend	see exercise 1
`helm install otel-collector open-telemetry/opentelemetry-collector`	Installs the OTel Collector	see exercise 2
`OTEL_EXPORTER_OTLP_ENDPOINT`	Env var telling an app where to send OTLP telemetry	`http://otel-collector:4317`
`OTEL_SERVICE_NAME`	Env var naming the service in its spans	`orders`
`traceparent` header	W3C Trace Context — carries `trace_id`/`span_id` between services	(HTTP header)
UI: Grafana Explore → Tempo → Search	Find and view traces (waterfall)	exercise 4

Collector config anatomy (the three sections you'll edit in exercise 2):

receivers:            # how telemetry gets IN
  otlp:
    protocols:
      grpc: {}        # listen for OTLP/gRPC on :4317
      http: {}        # and OTLP/HTTP on :4318
processors:           # what happens in the MIDDLE
  batch: {}           # group spans before export (efficiency)
exporters:            # where telemetry goes OUT
  otlp/tempo:
    endpoint: tempo:4317
    tls: {insecure: true}
service:
  pipelines:
    traces:           # wire receivers -> processors -> exporters
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

receivers.otlp.protocols.grpc/http — the Collector accepts OTLP on 4317 (gRPC) and 4318 (HTTP); your apps point OTEL_EXPORTER_OTLP_ENDPOINT here.
processors.batch — batches spans to reduce export overhead; add tail_sampling here for smart sampling.
exporters.otlp/tempo.endpoint — where processed traces are shipped (Tempo). Swap this one line to send to Jaeger instead — the decoupling payoff.
service.pipelines.traces — nothing is active until it's wired into a pipeline; a receiver/exporter defined but not referenced here does nothing (the ServiceMonitor-style "declared but not selected" trap, again).

Hands-on exercises

Continue on the kind cluster with kube-prometheus-stack, Grafana, and Loki from modules 01-04.

(WSL2) Install Tempo.

helm repo add grafana https://grafana.github.io/helm-charts   # (already added in module 04)
helm repo update
helm install tempo grafana/tempo --namespace monitoring
kubectl get pods -n monitoring | grep tempo

Expect a tempo pod Running. This is your trace store — the Loki-of-traces.

(WSL2) Install the OpenTelemetry Collector.

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace monitoring \
  --set mode=deployment \
  --set image.repository=otel/opentelemetry-collector-contrib \
  --set-string 'config.exporters.otlp/tempo.endpoint=tempo:4317' \
  --set-string 'config.exporters.otlp/tempo.tls.insecure=true' \
  --set 'config.service.pipelines.traces.exporters={otlp/tempo}'
kubectl get pods -n monitoring | grep otel-collector

(If the --set overrides are fiddly on your Helm version, install with a -f values.yaml file containing the Collector config from the Command reference — the point is a Collector receiving OTLP on 4317 and exporting to Tempo.) Verify the pod is Running and check its logs for "Everything is ready".

(WSL2) Deploy two services that call each other, auto-instrumented. Use the OpenTelemetry demo's simplest pair, or this minimal two-hop setup where a frontend calls a backend, both auto-instrumented to emit OTLP to the Collector. The essential wiring on each Deployment is the env:
```
#   OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector-opentelemetry-collector:4317
#   OTEL_SERVICE_NAME=frontend   (and =backend on the other)
```
Deploy both into the demo namespace, expose the frontend, and drive traffic through it so the frontend calls the backend on each request. (Any pre-instrumented sample two-service app works; the OpenTelemetry Community "astronomy shop" demo is the canonical one but heavy — a two-pod HTTP caller/callee is enough.)
(WSL2) View your first trace. Add Tempo as a Grafana data source (Connections → Data sources → Tempo, URL http://tempo:3100). In Explore, pick Tempo, click Search, and open a recent trace. You should see a waterfall: a root span from frontend with a child span from backend nested under it — one request, two services, joined. That join only happened because the trace context propagated.
Diagnose and fix: a missing span because context wasn't propagated. This is the tracing bug. Reproduce it: on the frontend, make it call the backend in a way that doesn't forward the traceparent header — e.g. set OTEL_PROPAGATORS=none on the frontend, or (if manually instrumented) have it build the outbound request without injecting context. Drive traffic, then search Tempo again. Now you see two separate traces — a frontend root with no child, and a backend root standing alone — instead of one joined trace. Diagnose: inspect the outbound request and confirm no traceparent header is present (the backend had nothing to continue, so it started a new root). Fix by restoring propagation (remove OTEL_PROPAGATORS=none / re-enable context injection); confirm the trace rejoins into one waterfall with the backend span nested under frontend. Lesson: a broken trace is almost always a propagation problem — one hop didn't pass traceparent.
(WSL2) Inspect span attributes. Open a joined trace and expand the spans. Note the attributes auto-instrumentation attached: http.method, http.route, http.status_code, durations. These are what let you filter traces ("show me traces where http.status_code=500") — the trace pillar's version of structured detail.
(WSL2) Add a manual span (if your sample app is one you can edit). Wrap a bit of "business logic" in a manual span with a custom attribute (e.g. order.value). Re-run and find it nested in the waterfall. This is the auto+manual combination — auto for the HTTP plumbing, manual for the domain-specific step that explains your system.
(WSL2) Understand sampling. In the Collector config, note there's no tail-sampling processor yet, so you're keeping everything (fine at this volume). Read the processors section and identify where a tail_sampling processor (keep 100% of errors, 10% of the rest) would go and why that belongs in the Collector, not each app. Write one sentence on why a specific request might have no trace once sampling is on (module 07 gotcha).
(WSL2) Note the Jaeger alternative. You used Tempo for the one-Grafana payoff. In a sentence, note when you'd pick Jaeger instead (you want its standalone, tracing-specialized UI, or you're in an ecosystem already standardized on it) — the same "know the alternative" discipline as ELK in module 04.
(WSL2) Leave Tempo and the Collector installed for module 07. To reclaim: helm uninstall tempo otel-collector -n monitoring.

Independent challenge

No full app given — assemble it yourself using this module plus module 04 (structured logging) and module 00 (why trace_id is high-cardinality). Stand up a two-service request path of your choosing (frontend → backend, or api → worker) where a request to the first triggers a call to the second, instrument both with OpenTelemetry pointing at your Collector, and prove you get one joined trace spanning both services with the child span nested correctly. Then do the hard part: make each service also log (structured JSON) the trace_id of the request it's handling, so the same identifier appears in both the trace and the logs. This is the deliberate setup for module 07's correlation — verify by picking a trace_id from a Tempo waterfall and finding that exact ID in the Loki logs of both services. The skill is producing telemetry that's already correlatable across pillars, which is 90% of what makes an incident debuggable.

Stuck? One hint

The join depends entirely on the traceparent header surviving the frontend→backend hop — if you get two disconnected traces, propagation is off (exercise 5). To get the trace_id into your logs, most OTel SDKs expose the current span context to your logging code; pull span.get_span_context().trace_id (or your language's equivalent) and add it as a trace_id field in the structured log line. Then in Loki, {namespace="demo"} | json | trace_id="<the-id-from-tempo>" should return lines from both services — trace_id lives in log content, never a Loki label (module 04's cardinality rule), because it's as high-cardinality as identifiers get.

Common mistakes & troubleshooting

Broken trace = broken propagation. Two disconnected traces where you expected one almost always means a hop didn't forward traceparent. Check the outbound request headers first; suspect OTEL_PROPAGATORS, a hand-built HTTP client that skips injection, or a proxy stripping the header.
Nothing in Tempo at all. Check the chain end to end: app env OTEL_EXPORTER_OTLP_ENDPOINT points at the Collector's 4317; the Collector's traces pipeline actually references the OTLP receiver and the Tempo exporter (declared-but-not-wired does nothing); Tempo is reachable from the Collector.
trace_id in a metric or Loki label. It's the highest-cardinality value there is — one per request. It belongs in trace data and in log content, never an indexed label (modules 00/04).
Expecting every request to have a trace. Once sampling is on, a specific request may legitimately have no stored trace. That's sampling working, not a bug — but it's a real surprise when correlating (module 07).
Only auto-instrumenting. Auto covers HTTP/DB plumbing but not your business logic; a trace with no domain spans can't explain why your specific operation was slow. Add manual spans for the steps that matter.
Sending traces straight to the backend, skipping the Collector. Works for a demo, but you lose central sampling/cost control and couple every app to the backend. Route through the Collector.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

Show questions

Define span and trace, and describe what a trace waterfall shows you that metrics and logs can't.
What is trace context propagation, which header carries it, and what happens if one service fails to propagate it?
Auto- vs. manual instrumentation — what does each cover, and why do real systems use both?
What are the three sections of an OTel Collector pipeline, and what's the benefit of routing through a Collector instead of app→backend directly?
Head vs. tail sampling — the difference, and why tail sampling is smarter for catching problems?
Why might a specific request you're investigating have no stored trace, and is that a bug?
Where does a trace_id belong across the three pillars, and where must it not go?

Show answers

A span is one timed operation (name, start/end, attributes); a trace is the tree of spans for a single request across services. The waterfall shows where the time went — which service/hop of one request was slow or failed — which an aggregate metric (not one request) and a single log line (one point, not the path) can't show.
Propagation is passing the trace context (trace_id + current span_id) from one service to the next, carried in the W3C traceparent HTTP header. If a service fails to propagate it, the trace breaks in two — the downstream service starts a new root span instead of continuing the trace.
Auto-instrumentation covers common framework plumbing (HTTP servers/clients, DB drivers) and propagation with little code change; manual instrumentation adds spans around your own business logic with custom attributes. Both, because auto gives fast coverage of the plumbing while manual explains the domain-specific steps auto can't see.
receivers (how telemetry gets in — OTLP), processors (batching, sampling, attributes), exporters (where it goes — Tempo/Jaeger), wired together in a service.pipelines entry. Routing through the Collector decouples apps from the backend, centralizes sampling/cost control, and can handle all three signals in one place.
Head sampling decides at the start of the trace (cheap, random, may drop the interesting ones); tail sampling decides after the trace completes, so it can keep 100% of errored/slow traces and a small fraction of normal ones — better for catching problems, at higher cost, done in the Collector.
Because sampling deliberately keeps only a fraction of traces, so that particular request may not have been retained. It's expected behavior, not a bug — but a real gotcha when you try to correlate a specific request in module 07.
In the trace data and in log content (as a field). It must not be a metric label or a Loki index label — it's the highest-cardinality value there is (one per request) and would explode either index.

Cumulative review

Closed book. Cover the answers and write each out before checking. These mix everything so far across all three pillars — concepts (00), Prometheus/PromQL (01-02), Grafana (03), Loki (04), and tracing (05).

For an e-commerce checkout, name one signal you'd collect from each pillar, and for each state whether order_id should appear in it and why.
A histogram_quantile p95 panel in Grafana shows "No data". Give two independent causes — one PromQL (module 02), one Grafana/label (module 03) — and how you'd tell them apart.
You put user_id as a label in three places by mistake: a Prometheus metric, a Loki stream label, and… where does it actually belong? Explain the single underlying rule that governs all three.
Write the LogQL to compute the per-service error-line rate over 5m from JSON logs, and say why this can sit on the same Grafana dashboard as a Prometheus panel.
A trace shows the frontend span with no backend child, and separately a lone backend trace. What's the cause and the fix?
Distinguish up == 0 (module 01), a Loki empty-selector result (module 04), and a broken trace (module 05) — what do these three "nothing showed up" failures have in common conceptually?
Why do you wrap a counter in rate() but read a gauge directly, and give one example metric of each type from the stack you installed.
An on-call engineer sees a latency-spike metric alert. Describe the ideal metric → logs → trace path they'd follow and which tool serves each step.
You need to keep 100% of errored requests' traces but only 5% of normal ones, without your app code deciding. What feature, in which component, achieves this?
Name one thing Container Insights/Log Analytics (track 07) did for you automatically that you had to configure by hand in this track, and one capability this open-source stack has that the Azure-native monitoring barely offered.