Reliability Patterns

Why this matters

At-least-once delivery (modules 00, 02, 04) is a promise with sharp edges: messages retry, duplicate, get stuck, and fail in ways your happy path never sees. This module is the toolbox that turns that promise into a system you can run: retry with backoff so transient failures self-heal instead of hammering a struggling dependency, dead-lettering so unprocessable messages are quarantined not lost, the outbox pattern so a database write and the event announcing it can't disagree, and poison-message handling so one bad message doesn't wedge a whole consumer. These are the patterns behind every "proven dead-letter handling" and "idempotent consumer" line in the capstone.

Cost warning: exercises use a Service Bus Standard namespace and, optionally, a Container Apps Dapr consumer (module 04). Both are cheap at lab volume, but a retry storm (a consumer failing fast in a tight loop, or Dapr retrying aggressively) generates real per-operation charges and log volume — fix failing consumers promptly and clean up at the end.

Concepts

Retry with backoff (and jitter)

When a consumer fails to process a message because a dependency is temporarily unavailable (a database blip, a rate limit, a downstream 503), the right move is to retry — but how you retry matters. Immediate, unbounded retries turn a struggling dependency into a downed one (a self-inflicted DDoS) and burn per-operation cost. The pattern is exponential backoff: wait 1s, then 2s, 4s, 8s… up to a cap, so pressure on the dependency eases as it struggles, plus jitter (a random spread) so a thousand consumers that failed at the same instant don't retry in a synchronized thundering herd. Crucially, distinguish transient failures (retry — the dependency will likely recover) from permanent ones (a malformed message will never succeed no matter how many retries — that's a poison message, below). In Service Bus, retry is driven by not-completing so the message is redelivered; in Dapr, by resiliency policies (retry/backoff/circuit-breaker config) applied to the component. The skill is bounding retries so a permanent failure eventually stops retrying and gets dead-lettered instead of looping forever.

Dead-lettering as a deliberate design, not an accident

You met the DLQ in module 02 (Service Bus) and Event Grid's dead-letter storage in module 03. The pattern-level point: dead-lettering is where messages go to be seen, not to disappear. A message should be dead-lettered when retries are exhausted (transient failure that never cleared) or immediately when it's recognized as unprocessable (a poison message). A well-designed system treats the DLQ as an operational surface: its depth is monitored and alerted (module 07), each dead-lettered message carries a reason (DeadLetterReason, DeadLetterErrorDescription) explaining why, and there's a defined redrive path — inspect, fix the root cause, and either resubmit the message to the main entity or discard it deliberately. "Set MaxDeliveryCount and never look at the DLQ" is the anti-pattern that quietly loses business events (module 02's diagnose-and-fix). Dead-lettering is only reliable if someone (or an alert) is watching the letterbox.

The outbox pattern: consistency between a DB write and an event publish

Here's the trap at the heart of event-driven systems. Your service needs to do two things atomically: write to its database (order saved) and publish an event (OrderPlaced). But the database and the message broker are two different systems — there's no shared transaction. If you write the DB then publish and crash in between, the order exists but no one is notified (a lost event). If you publish then write and crash between, consumers react to an order that doesn't exist (a phantom event). This is the dual-write problem, and it has no solution as long as you try to write both directly.

The outbox pattern solves it: within the same database transaction that saves the order, also insert the event into an outbox table in that same database. That single local transaction is atomic — either both the order and the outbox row commit, or neither does. A separate relay process (a poller, or a change-data-capture stream off the DB log) then reads unpublished outbox rows and publishes them to the broker, marking each published. Because the relay publishes at-least-once (it may crash after publishing but before marking, and re- publish), consumers must be idempotent (module 00) — the outbox guarantees the event is never lost, and idempotency handles the duplicate. Outbox + idempotent consumer is the workhorse recipe for reliable event-driven services.

Poison messages and how to stop one wedging a consumer

A poison message is one that can never be processed successfully — malformed JSON, a reference to a deleted entity, a schema the consumer doesn't understand (module 06). The danger: a naive consumer receives it, fails, doesn't complete it, receives it again (at-least-once), fails again… and if it's at the head of an ordered/session queue, it blocks every message behind it — a single bad message wedges the whole consumer. The defenses stack:

• Bound the retries (MaxDeliveryCount): after N attempts the broker auto-dead-letters it, so it stops blocking (module 02).
• Detect and dead-letter immediately: if the consumer can tell a message is permanently bad (parse error, validation failure), explicitly dead-letter it on the first attempt with a clear reason — don't waste N retries on something that will never succeed.
• Catch-and-quarantine in code: wrap processing so an unexpected exception dead-letters (or routes to an error topic) with context, rather than throwing in a way that just abandons and re-loops.

The capstone requires you to force a poison message and prove your dead-lettering actually catches it — because a dead-letter path you've never tested is a dead-letter path that doesn't work.

Putting it together: the reliable consumer shape

A production consumer composes all of the above with the idempotency from module 00 into one shape: (1) receive; (2) check the dedup key — already processed? complete and return (idempotent no-op); (3) validate — malformed? dead-letter immediately with a reason (poison); (4) do the side effect and record the dedup key in one transaction (and if this service also emits events, write them to the outbox in that same transaction); (5) on transient failure, let it retry with backoff up to a bound; (6) on exhausted retries, it dead-letters automatically. Every element of this shape is one of the patterns above, and this is essentially the consumer the capstone asks you to design and defend.

Command reference

Field-by-field — a Dapr resiliency policy applied to the pub/sub component:

# resiliency.yaml (Dapr)
spec:
  policies:
    retries:
      pubsubRetry:
        policy: exponential          # exponential backoff, not constant
        maxInterval: 30s             # cap the backoff growth
        maxRetries: 5                # bound it — then it stops (→ dead-letter)
    circuitBreakers:
      pubsubCB:
        maxRequests: 1
        trip: consecutiveFailures >= 5   # stop hammering a dead dependency
  targets:
    components:
      orderpubsub:
        inbound:
          retry: pubsubRetry
          circuitBreaker: pubsubCB

• policy: exponential — exponential backoff (vs. constant), the pattern from Concepts; add spacing that grows with each failure.
• maxInterval: 30s — caps how large the backoff delay can grow.
• maxRetries: 5 — bounds retries so a permanent failure eventually stops retrying (and the message can be dead-lettered) instead of looping forever.
• circuitBreaker: trip: consecutiveFailures >= 5 — after repeated failures, stop sending to the dependency for a cooldown, so you don't hammer something that's down (the retry-storm defense).

Flag breakdown — az servicebus queue create ... --default-message-time-to-live PT1H --dead-lettering-on-message-expiration true:

• --default-message-time-to-live PT1H — messages live 1 hour (ISO-8601 duration); after that they're expired.
• --dead-lettering-on-message-expiration true — expired messages go to the DLQ (with reason TTLExpiredException) instead of vanishing — so "too old to be relevant" is still observable, not silently lost.

Hands-on exercises

These build on the Service Bus namespace (module 02) and, optionally, the Dapr consumer (module 04). Use the portal Explorer to send/receive where no code is implied. Cleanup is last.

1. Set up.

   az group create -n rg-rel -l eastus
   az servicebus namespace create -g rg-rel -n sb-rel-$RANDOM --sku Standard
   az servicebus queue create -g rg-rel --namespace-name <ns> -n work --max-delivery-count 3
   

2. Watch bounded retry → auto-dead-letter. Send a message to work. In the Explorer, Receive-in-PeekLock and Abandon it repeatedly (simulating transient failure).

   az servicebus queue show -g rg-rel --namespace-name <ns> -n work --query countDetails
   

   Verify: after 3 delivery attempts the message leaves activeMessageCount and appears in deadLetterMessageCount — bounded retry prevented an infinite loop. Peek the DLQ and read DeadLetterReason = MaxDeliveryCountExceeded.

3. Dead-letter on TTL expiry. Create a queue with a short TTL and DLQ-on- expiry:

   az servicebus queue create -g rg-rel --namespace-name <ns> -n perishable --default-message-time-to-live PT1M --dead-lettering-on-message-expiration true
   

   Send a message and don't consume it for over a minute.

   Verify: it moves to the DLQ with reason TTLExpiredException — "stale" is now observable, not silently gone.

4. Immediate dead-letter of a poison message. In the Explorer, Receive a message from work in PeekLock and use Dead-letter directly (with a reason like MalformedPayload), rather than abandoning it repeatedly.

   Verify: it lands in the DLQ on the first attempt with your reason — the "detect and dead-letter immediately" defense, sparing 3 wasted retries on something that would never succeed.

5. Prove a poison message wedges an ordered consumer (then unwedge it). Create a session-enabled queue (module 02), send session A: a poison message A1 followed by good A2, A3. Accept the session and try to process in order.

   Observe: A1 fails repeatedly and, because a session locks the ordered stream to one consumer, A2/A3 are blocked behind it. Fix: bound delivery (--max-delivery-count) so A1 auto-dead-letters after N attempts and the session proceeds to A2 — or detect and dead-letter A1 immediately. The lesson: in ordered processing, a poison message is a head-of-line blocker; bounded retry / immediate dead-letter is the unwedging mechanism.

6. Design the outbox on paper, then simulate it. For a service that saves an order and publishes OrderPlaced: (a) write the failure timeline for the naive "save then publish" if it crashes in between, and for "publish then save"; (b) design the outbox — the single transaction that writes both the order and an outbox row, plus the relay that publishes unmarked rows. Then simulate the relay: insert two rows into a mock "outbox" (e.g. two messages staged in a table or file), publish them to work via the Explorer, and mark them published.

   Verify: your timeline shows the naive approaches produce a lost or phantom event, while the outbox's single local transaction cannot — and the relay's re-publish-on-crash is why the consumer must be idempotent.

7. Add Dapr resiliency (optional, if you did module 04). Register the resiliency.yaml policy from the Command reference against your pub/sub component, make the consumer fail transiently, and observe the backoff spacing in the timing of redeliveries (and the circuit breaker tripping after repeated failures).

   Verify: redelivery intervals grow (exponential) rather than hammering instantly, and after the trip threshold the consumer stops being called for a cooldown — retry-storm avoided.

8. Diagnose and fix: a DLQ quietly filling up unnoticed. Over exercises 2-5 you've accumulated dead-lettered messages nobody is draining. Read the metric the way an alert would:

   $nsId = az servicebus namespace show -g rg-rel -n <ns> --query id -o tsv
   az monitor metrics list --resource $nsId --metric DeadletteredMessages --interval PT1M -o table
   

   Observe: a non-zero, growing dead-letter count no process is handling. Diagnose: dead-lettering worked, but there's no monitoring/redrive — the failure quarantine has no operator. Fix: define the redrive path (peek the DLQ, read reasons, fix root cause, resubmit or discard) and — the durable fix — an alert on DeadletteredMessages > 0 (or a threshold). This is the exact metric you'll formalize in module 07.

9. Diagnose and fix: a retry storm from unbounded, no-backoff retries. Scenario: a consumer with max-delivery-count effectively unbounded (very high) and zero backoff hammers a downstream that's returning 503, driving Service Bus operation counts (and cost) through the roof while never making progress.

   Diagnose: retries are unbounded and instantaneous, converting a transient downstream failure into a self-inflicted load + cost problem. Fix: bound max-delivery-count (so it dead-letters and stops) and add exponential backoff + a circuit breaker (Dapr resiliency, or backoff in consumer code) so pressure eases instead of compounding. Note the cost angle: every retry is a billable operation.

10. Clean up.

    az group delete -n rg-rel --yes --no-wait
    

    Verify: az servicebus namespace list -o table no longer lists the namespace once deletion completes.

Independent challenge

Design and implement (as far as the portal/CLI allow without full app code) a reliable consumer for an orders topic that composes every pattern in this module: bounded retry with backoff, immediate dead-lettering of a deliberately malformed poison message, a monitored DLQ with a written redrive plan, and — on paper with a simulated relay — an outbox that keeps a database write and the OrderPlaced publish consistent. Force at least one poison message and prove it ends up in the DLQ with a meaningful reason, and force a transient failure and show it recovers via retry rather than dead-lettering. Write up the final consumer shape (the six-step sequence from Concepts) and mark exactly where idempotency (module 00) and the outbox each fit. This draws on modules 00, 02, and 04. Tear down the namespace when finished — retries and dead-letter accumulation are billable operations, and a forgotten Standard namespace keeps counting them.

Common mistakes & troubleshooting

• Unbounded or no-backoff retries. Immediate infinite retries turn a transient failure into an outage and a cost spike. Always bound retries (max-delivery-count) and space them (exponential backoff + jitter).
• Retrying a permanent failure. A malformed message will never succeed; retrying it N times just delays the inevitable dead-letter and wastes operations. Detect permanent failures and dead-letter immediately.
• Treating the DLQ as a black hole. Dead-lettering only helps if the DLQ is monitored and has a redrive path. An unwatched DLQ is silent data loss.
• The dual-write trap. Writing to the DB and publishing to the broker as two independent operations will eventually lose or phantom an event on a crash. Use the outbox (single local transaction + relay).
• Assuming the outbox gives exactly-once. The relay re-publishes on crash; it's at-least-once. Consumers still need idempotency.
• Untested dead-letter paths. A dead-letter route you've never exercised often doesn't work (wrong reason capture, no destination, silent drop — Event Grid module 03). Force a poison message and verify, as the capstone requires.
• Cost pitfall — retry storms and log volume. Every retry is a billable Service Bus operation and a log line; a tight failing loop multiplies both. Bounded backoff and circuit breakers cap the blast radius. Also delete idle Standard namespaces — they keep metering per-operation traffic (including your own diagnostic polling).

Cumulative review

Closed-book. Cover the answers and write each one out before checking — this mixes everything from modules 00-05.

1. Distinguish transient from permanent processing failures, and give the correct reliability response to each.
2. Explain the dual-write problem in one or two sentences, and how the outbox pattern eliminates it. Why does it still require idempotent consumers?
3. A poison message sits at the head of a session-ordered queue. What happens to the messages behind it, and what two mechanisms unwedge the consumer?
4. In Service Bus PeekLock, what actually triggers a redelivery, and what bounds the redeliveries so a message can't loop forever?
5. You're choosing between backing a Dapr pub/sub topic with Service Bus vs. calling the Service Bus SDK directly. Give one reason for each choice, tying to the abstraction trade-off from module 04.
6. Contrast point-to-point and publish/subscribe, then name the Azure entity for each and how Dapr's consumerID selects between the two behaviors.
7. Why does an event-driven system prefer eventual consistency, and what two reliability patterns are what actually make that eventual consistency converge rather than silently drop events?
8. An Event Grid subscription's handler is down. What does Event Grid do, and what must be configured so events aren't lost — and how does that compare to a Service Bus DLQ?
9. Write the six-step shape of a reliable consumer and label which step handles duplicates, which handles poison messages, and which handles a downstream dependency being temporarily down.
10. Your Service Bus bill jumped and a queue's operation count is enormous though nothing is getting processed. What's the most likely cause and the fix?

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. Why is exponential backoff with jitter better than immediate constant retries, in terms of both the dependency and cost?
2. Give the two ways a message ends up dead-lettered and the two-part definition of a "well-designed" dead-letter setup.
3. State the dual-write problem and how the outbox pattern removes it.
4. Why does the outbox pattern still require idempotent consumers?
5. What is a poison message, why is it especially dangerous in an ordered/ session queue, and name two defenses.
6. In Dapr, which construct configures retry/backoff/circuit-breaker behavior, and what does maxRetries protect you from?
7. Walk the six-step shape of a reliable consumer.
8. A consumer is stuck in a tight retry loop hammering a 503-ing downstream and your bill is climbing. Name the two fixes.

Next

06-event-schemas-and-versioning — the poison-message-by-schema problem from this module, addressed at the source: why event schema matters more in async systems than in synchronous APIs, how to evolve a schema without breaking every consumer at once, and a lightweight schema-registry concept.