Observability for Event-Driven Systems

Why this matters

Every diagnose-and-fix in this track — the stuck message, the silently filling DLQ, the filter that never matches, the duplicate side effect — had the same root problem: nobody could see it until it hurt. Synchronous systems are easy to observe because a failure is a response you get back; event-driven systems hide failure inside brokers and across hops where no single request ties it all together. This module is the fix: tracing one message end-to-end across queue hops (building directly on the distributed tracing you set up in track 12) and treating queue depth and consumer lag as first-class metrics you alert on, so the failures earlier modules made you hunt for announce themselves instead.

Cost warning: exercises use a Service Bus Standard namespace and Azure Monitor metrics (metrics on the resource are free to read; Log Analytics ingestion for diagnostic logs and Application Insights traces bills per GB ingested). Lab volumes are negligible, but a chatty trace/log firehose left running is a real ingestion bill — keep sampling sane and clean up at the end.

Concepts

Why async observability is different: no single request to follow

In a synchronous call chain (track 12) a trace follows one thread of execution: A calls B calls C, and the whole span tree lives inside one request's lifetime. Event-driven systems shatter that. A publish returns immediately — the publisher is done and gone before the consumer even receives the message; the consumer might run seconds or minutes later, in a different process, possibly after a redelivery or a DLQ round-trip. There is no ambient request linking publish to consume. So the three questions you actually ask in production — where is this message now? why is this one slow? why did this one fail? — can't be answered by request logs alone. You need two things async-specific: trace context propagated through the message itself (so the consumer's work is stitched to the publisher's), and broker-level metrics (depth, lag, DLQ count) that describe the queue, not any one request. The rest of this module is those two things.

Propagating trace context across a queue hop

Distributed tracing works by every span carrying a trace context (a trace id plus the parent span id) — in track 12 that was the W3C traceparent header flowing over HTTP. Across a queue hop there's no HTTP call to carry it, so the pattern is: the publisher writes the trace context into the message (as an application property / message header, e.g. Diagnostic-Id or a traceparent property), and the consumer reads it back out and starts its processing span as a child (or a link) of the publisher's span. Now a single trace shows publish → [time in queue] → receive → process, spanning both services and the broker wait in between. The good news: you often get this for free. Application Insights/OpenTelemetry auto-instrumentation for Service Bus injects and extracts this context automatically, and Dapr propagates W3C trace context through its pub/sub building block (module 04) out of the box — so the Dapr consumer you built is already stitching publisher and consumer traces if tracing is enabled on the environment. The concept to hold: the message is the transport for trace context when there's no request to ride on.

Queue depth as a first-class metric

The single most important operational number in a messaging system is queue depth — the count of active, unprocessed messages waiting (activeMessageCount, the counter you've watched all track). It's a direct, leading indicator of health: a depth that hovers near zero means consumers keep up; a steadily rising depth means consumers are falling behind (down, too few, too slow, or wedged on a poison message from module 05) before users notice a delay. Two companions matter as much: dead-letter count (deadLetterMessageCount) — anything above zero is failed business events needing attention (module 02/05's silently-filling-DLQ bug is exactly an unmonitored version of this), and incoming vs. outgoing rate — if the publish rate exceeds the completion rate, depth will grow, so comparing the two predicts a backlog before depth itself climbs. These are entity-level metrics Azure Monitor exposes for Service Bus (ActiveMessages, DeadletteredMessages, IncomingMessages, OutgoingMessages); depth and DLQ are the two you alert on first.

Consumer lag: the stream analogue, and why it's not the same as depth

For streams (Event Hubs/Kafka, module 01) there's no "depth" in the queue sense — messages aren't removed on read, they're retained and each consumer holds an offset. The health metric there is consumer lag: how far a consumer's offset trails the latest event in the partition — i.e. how many events behind it is. Lag is the stream-native "am I keeping up?" number, and it behaves differently from queue depth: depth is shared (one backlog all consumers of a queue drain), while lag is per consumer group (a slow analytics consumer can lag badly while a fast alerting consumer stays current on the same stream). Growing lag means the same thing growing depth does — the consumer isn't keeping up — but you can also have lag on a healthy system that simply hasn't caught up after a deploy. The takeaway: for queues, watch depth + DLQ; for streams, watch per-consumer-group lag, and in both cases the alertable signal is sustained growth, not any instantaneous value.

Correlation, dashboards, and closing the loop with earlier modules

Observability pays off when the pieces connect. A correlation id — a business key (order id) or the envelope id you already carry for idempotency (module 00) and versioning (module 06) — lets you pull every log line and span for one logical event across every hop, so "what happened to order-42?" is a single query, not a cross-team scavenger hunt. A useful event-driven dashboard layers: depth + DLQ per entity (are we keeping up / are things failing?), end-to-end latency from the propagated traces (publish-to-process time, including queue wait), and throughput (incoming vs. outgoing rate). And it closes the loop on the whole track: an alert on DLQ depth > 0 would have caught module 02/05's quiet failures; a depth-rising alert catches the stuck/wedged consumer from module 02/05; a trace across the hop catches the "which service dropped it?" question the outbox and dead-letter paths (module 05) raise. Observability isn't a separate concern bolted on — it's how every reliability pattern in this track becomes operable.

Command reference

Flag-by-flag — az monitor metrics list --resource <ns-id> --metric ActiveMessages --dimension EntityName --interval PT1M --aggregation Average:

• --resource <ns-id> — the Service Bus namespace resource id whose metrics you want.
• --metric ActiveMessages — the metric = queue depth (also DeadletteredMessages, IncomingMessages, OutgoingMessages).
• --dimension EntityName — split the metric per queue/topic so you see which entity is backing up, not just a namespace total.
• --interval PT1M — one-minute buckets (ISO-8601), fine-grained enough to see a backlog forming.
• --aggregation Average — how to roll up within a bucket (Average/Maximum/ Total); use Maximum for depth to catch spikes.

Flag-by-flag — az monitor metrics alert create -g rg-obs -n dlq-not-empty --scopes <ns-id> --condition "total DeadletteredMessages > 0" --window-size 5m --evaluation-frequency 1m --description "Dead-letter queue is not empty":

• --scopes <ns-id> — the resource the alert watches (the namespace).
• --condition "total DeadletteredMessages > 0" — the rule: any dead-lettered message trips it — the durable fix for the silently-filling-DLQ bug from modules 02/05.
• --window-size 5m / --evaluation-frequency 1m — evaluate every minute over a 5-minute window, so a transient blip doesn't page but a persistent problem does.
• --description — what the on-call human sees; make it actionable ("check the DLQ reason and redrive").

Hands-on exercises

These use a Service Bus Standard namespace and Azure Monitor. Metrics reads are free; if you wire Log Analytics/App Insights, keep volumes tiny. Cleanup is last.

1. Set up.

   az group create -n rg-obs -l eastus
   az servicebus namespace create -g rg-obs -n sb-obs-$RANDOM --sku Standard
   az servicebus queue create -g rg-obs --namespace-name <ns> -n work --max-delivery-count 3
   

2. Read depth as a point-in-time value. Send a few messages to work in the Explorer without consuming them, then:

   az servicebus queue show -g rg-obs --namespace-name <ns> -n work --query countDetails
   

   Verify: activeMessageCount equals the number you sent — this is queue depth, your primary health number. Consume them and confirm it drops to 0.

3. Read depth as a time series. Send messages, then pull the metric:

   $nsId = az servicebus namespace show -g rg-obs -n <ns> --query id -o tsv
   az monitor metrics list --resource $nsId --metric ActiveMessages --dimension EntityName --interval PT1M --aggregation Maximum -o table
   

   Verify: the work entity shows a non-zero max active-message count in the buckets where messages were waiting. This time series — not the instant value — is what an alert watches.

4. Watch depth rise under a slow consumer. Send 20 messages quickly, then consume them one at a time slowly (or not at all). Re-read the metric over a few minutes.

   Verify: depth climbs while publishing outpaces consumption and falls as you drain it. A sustained rise (publish rate > completion rate) is the leading indicator that consumers can't keep up — before any user sees a delay.

5. Compare incoming vs. outgoing rate.

   az monitor metrics list --resource $nsId --metric IncomingMessages,OutgoingMessages --interval PT1M --aggregation Total -o table
   

   Verify: in the window where you published faster than you consumed, IncomingMessages exceeds OutgoingMessages — the arithmetic reason depth grew. This comparison predicts a backlog before depth itself is alarming.

6. Alert on a non-empty dead-letter queue. Force a message to the DLQ (abandon past max-delivery-count, as in module 05), then create the alert:

   az monitor metrics alert create -g rg-obs -n dlq-not-empty --scopes $nsId `
     --condition "total DeadletteredMessages > 0" --window-size 5m --evaluation-frequency 1m `
     --description "Service Bus DLQ is not empty - inspect reason and redrive"
   

   Verify: the alert rule exists (az monitor metrics alert list -g rg-obs -o table) and, with a message in the DLQ, evaluates to fired. This single alert is what turns modules 02/05's "silently filling DLQ" from an incident into a page.

7. Propagate and read trace context across a hop (Dapr path). If you have the module 04 Dapr consumer and tracing enabled on the Container Apps Environment (track 12), publish an event and open the end-to-end transaction in Application Insights.

   Verify: a single trace spans publish → queue wait → consumer /orders handler because Dapr propagated W3C trace context through the message. Note the queue-wait time as its own segment — latency you can't see without cross-hop tracing.

8. Correlate everything about one event. Publish a message carrying a known business id (e.g. orderId = ORD-42) as an application property / in the payload, let it be processed (and optionally dead-lettered). Then query by that id across your logs/traces.

   Verify: you can retrieve every hop for ORD-42 from one correlation key — the same id your idempotent consumer (module 00) and versioning (module 06) already rely on. "What happened to this order?" becomes one query.

9. Diagnose and fix: a wedged consumer visible only as rising depth. Reproduce module 05's head-of-line block — a session/ordered queue stuck on a poison message — but this time find it through observability, not by knowing where to look. Watch ActiveMessages climb while OutgoingMessages sits near zero.

   Diagnose: depth rising + outgoing flat = consumer is running but not completing anything → it's wedged (poison message / never-acking, the module 02/05 bugs). Confirm by checking delivery counts and the DLQ. Fix: the module 05 remedy (bound delivery so the poison dead-letters, or fix the consumer) — but the observability lesson is that a depth-rising alert (ActiveMessages growing for N minutes) would have surfaced this automatically. Add that alert.

10. Diagnose and fix: an under-observed stream consumer (lag). Conceptually (or with an Event Hub from module 01): an analytics consumer group falls hours behind while an alerting consumer group on the same stream stays current, and nobody noticed because "the stream looks fine."

    Diagnose: depth is the wrong metric for a stream — messages aren't removed, so the queue "looks empty/healthy." The real signal is per- consumer-group lag (offset distance behind the latest event), which is per-group, so one group can lag badly while another is fine. Fix: monitor and alert on consumer lag per group, not stream depth; sustained lag growth is the keeping-up signal for streams.

11. Clean up.

    az group delete -n rg-obs --yes --no-wait
    

    Verify: az servicebus namespace list -o table no longer lists the namespace, and the alert rule is gone with the resource group. Also disable any diagnostic settings shipping to a Log Analytics workspace you want to stop billing.

Independent challenge

Instrument the reliable consumer you designed in module 05 so that every failure mode in this track would announce itself. Stand up a Service Bus topic with a consumer, and wire: (1) an alert on DLQ depth > 0 and an alert on rising active-message depth (consumers falling behind); (2) cross-hop trace propagation so a single trace covers publish → queue wait → process (use the Dapr path from module 04 + track 12 tracing, or inject a traceparent property manually and describe how the consumer would read it); and (3) a correlation key (the envelope id from modules 00/06) that retrieves every log/span for one logical event. Then deliberately trigger two of this track's failures — a wedged consumer (rising depth) and a poison message (DLQ) — and show that your alerts fire and your trace/correlation lets you find the cause in one query, without knowing in advance where to look. Write a short paragraph mapping each alert back to the specific earlier-module bug it would have caught. This draws on modules 00, 02, 04, 05 and track 12. Delete the resource group and disable any diagnostic/ App Insights ingestion when done — trace and log ingestion bill per GB.

Common mistakes & troubleshooting

• Relying on request logs alone. A publish returns before the consumer runs, so there's no single request to follow. Without trace context in the message and broker metrics, cross-hop failures are invisible.
• Not propagating trace context through the message. If the publisher doesn't inject and the consumer doesn't extract, publisher and consumer traces are two unrelated islands. Use Dapr/auto-instrumentation, or inject traceparent manually.
• Watching only instantaneous depth. A single high reading is noise; the alertable signal is sustained growth (or incoming rate persistently above outgoing). Alert on a window, not a point.
• Ignoring the DLQ metric. DeadletteredMessages > 0 is failed business events. The single highest-value alert in the whole track; without it, modules 02/05's silent DLQ recurs forever.
• Using queue-depth thinking for a stream. Streams retain messages, so depth "looks empty" while a consumer group lags for hours. Watch per-consumer-group lag for streams, not depth.
• Cost pitfall — trace/log ingestion volume. Metrics on the resource are free to read, but shipping diagnostic logs to Log Analytics and full traces to Application Insights bills per GB ingested — a high-throughput firehose with 100% trace sampling can dwarf your Service Bus cost. Sample sensibly, scope diagnostic categories, and disable ingestion you're not using. This is the same ingestion-cost discipline track 12 taught, applied to messaging telemetry.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. Why can't a normal request trace, on its own, follow a message through an event-driven system?
2. Across a queue hop there's no HTTP header to carry traceparent. How does trace context actually get from publisher to consumer, and what gives you this for free?
3. What is queue depth, why is it a leading indicator, and which companion metric is the highest-value one to alert on?
4. What's the difference between comparing incoming-vs-outgoing rate and just watching depth?
5. Why is "depth" the wrong health metric for a stream, and what do you watch instead — and why is that metric per-consumer-group?
6. You see active-message depth climbing while outgoing messages sit near zero. What does that combination tell you, and what earlier-module bug is it?
7. Name the single Service Bus alert that would have caught the "silently filling DLQ" bug from modules 02 and 05.

Next

08-capstone-project — put it all together: a Service Bus topic with filtered subscriptions, an Event Grid subscription reacting to a real Azure resource event, a Dapr pub/sub Container App consumer, proven dead-letter handling, an idempotent consumer design, and the depth/DLQ observability from this module — one working event-driven system, end to end.