02 - Designing and Testing a DR Plan

Why this matters

A DR plan that has never been executed is a hypothesis, not a plan. Module 01 gave you a substrate that can fail over; this module turns that capability into a written runbook a stressed on-call engineer can follow at 3am — and then, the part that actually separates real DR from theatre, has you execute a failover drill against your two regions and measure whether traffic truly redirected. Most "DR plans" die the first time they meet a real DNS TTL, a stale replica, or a step that assumed a human who's on holiday. You'll find those failures in a drill, on purpose, instead of during an outage.

Concepts

A runbook is written for the worst version of you

The audience for a DR runbook is a tired, stressed engineer, possibly not the author, during a real outage with alarms going off (the on-call context from track 20 / module 04). That dictates the form:

• Numbered, copy-pasteable steps, not prose. Each step is one action with the exact command and its expected output.
• Explicit preconditions and decision points — "if the primary's health endpoint returns non-200 for >2 min, declare a regional incident and proceed; otherwise investigate in-region first." Ambiguity is what causes hesitation during an incident.
• Verification after every consequential step, not just at the end — how do you know the database promoted, the traffic shifted, the app is serving? A runbook without verification steps is a wish list.
• Rollback / abort criteria — when do you stop and not fail over (because failing over unnecessarily is its own outage)?
• Named roles, not names — "the Incident Commander decides to declare," mapping to track 20 / module 05's incident-response roles, so the plan doesn't depend on one specific person being reachable.

The anatomy of a regional-failover runbook

A concrete structure you can reuse for any system:

1. Detection & declaration — what signal (SLO burn alert from track 20, health-probe failure, Azure Service Health) triggers this runbook, and who declares a DR event. Include the "don't fail over for a blip" guard.
2. Communication — declare an incident (track 20 / module 05), open the comms channel, set expectations. Failover is a customer-visible event.
3. Data promotion — promote the secondary region's data store to primary/writable (Azure SQL geo-failover, replica promotion, or storage account failover). This is usually the step with the real RPO consequence.
4. Traffic cutover — shift the global router to the secondary (disable the primary endpoint / raise the secondary's priority / let health probes do it). This is where DNS TTL bites.
5. Scale-up (pilot light / warm standby) — bring the secondary to full capacity if it wasn't already.
6. Verification — end-to-end checks that real user journeys work in the secondary, not just that pods are Running.
7. Failback plan — how you return to the primary once it recovers, which is often harder than the failover (you must re-replicate data that changed in the secondary without losing it).

Failover time is detection + promotion + cutover + warm-up

Your achieved RTO is the sum of the phases, and it's worth decomposing because each has a different lever:

RTO_achieved = t_detect + t_declare + t_promote_data + t_cutover(DNS) + t_scale_up + t_verify

• t_detect — health-probe interval × failures, or SLO burn-rate alert latency (track 20 / module 03).
• t_declare — human decision time; shrink with clear declaration criteria.
• t_promote_data — database/storage failover time (seconds to minutes).
• t_cutover(DNS) — the DNS-TTL term for Traffic Manager (near-zero for Front Door). This is the one people forget.
• t_scale_up — zero for warm standby / active-active, minutes for pilot light.
• t_verify — how long to confirm it actually works.

Measuring these in a drill is how you learn whether your paper RTO is achievable. Almost always the first drill blows the target, and the decomposed timing tells you which phase to fix.

DNS TTL and caching: why "failover" sometimes changes nothing

The single most common failover surprise (foreshadowed in module 01): you disable the primary endpoint, Traffic Manager immediately starts answering with the secondary IP, and yet users keep hitting the dead primary for minutes. Why? Every resolver and client between you and the user cached the old DNS answer for the record's TTL, and some (browsers, JVMs with infinite DNS caching, corporate resolvers that ignore low TTLs) hold it even longer. So the failover "succeeded" at the Traffic Manager level and did nothing observable for real users during the TTL window. The fixes, in order of preference:

1. Lower the TTL in advance (module 01's --ttl 30) — you can't lower it during an incident and have it help, because the high TTL is already cached; it must be low before the outage.
2. Use Front Door (L7 edge failover) if you need failover faster than any safe DNS TTL allows.
3. Know your worst-cached clients — some ignore TTL entirely; the runbook should set expectations that a small tail keeps hitting the old region.

You'll reproduce this exact trap in the drill and measure the window.

The drill is the deliverable — and its result is data

Echoing track 14 / module 04's "the restore is the deliverable": a DR plan is not done when it's written, it's done when a drill has executed it and you have recorded results: achieved RTO/RPO per phase, every step that was wrong/missing/ambiguous, and a list of fixes. A drill that "went fine" with no notes is nearly worthless — the value is in the discrepancies you find (a step that assumed a credential you didn't have, a replica that was 40 minutes behind, a scale-up that hit a quota). Record them, fix the runbook, and re-drill. This "execute and record, don't just write" discipline is the whole point of this module and the backbone of the game days in module 06.

Command reference

Flag breakdown — forcing a failover by disabling the primary endpoint (the controlled drill trigger — safer than actually killing the region):

az network traffic-manager endpoint update \
  -g dr-rg --profile-name app-tm -n primary \
  --type externalEndpoints \
  --endpoint-status Disabled

• --endpoint-status Disabled — Traffic Manager stops handing out this endpoint's IP, so new DNS answers point at the secondary. This simulates "the primary is out of rotation" without destroying the primary — you can re-enable it to fail back. Crucially it does not flush anyone's cached DNS, so it exposes the real TTL window.

Flag breakdown — promoting an Azure SQL failover group (data promotion step):

az sql failover-group set-primary \
  --name <fg-name> \
  --resource-group <secondary-rg> \
  --server <secondary-server>

• --server <secondary-server> — the server in the DR region you're making primary. Run against the secondary — you're telling Azure "this side is now authoritative." A forced failover (add --allow-data-loss in a real regional outage where the primary is unreachable) accepts the async replication gap as your realized RPO — measure it.

Hands-on exercises

These run a real drill against the two regions from module 01. Rebuild that substrate (or a minimal echo app on two clusters behind one Traffic Manager profile) first. Keep a timer/notes file open — the recorded results are the deliverable, not just "it worked." Tear the second region down in exercise 9.

1. Write the runbook before touching anything

Using the anatomy above, write an actual numbered runbook (in a dr-runbook.md) for "primary region is down, fail the web app to the secondary." Every step must have the exact command and an expected-output verification. Include declaration criteria and a failback outline. Do this first — the drill's job is to find where this written plan is wrong.

2. Establish a steady-state baseline

FQDN=$(az network traffic-manager profile show -g dr-rg -n app-tm --query dnsConfig.fqdn -o tsv)
dig +short "$FQDN"                       # should be the PRIMARY ip
curl -s "http://$FQDN/" | grep -o "Welcome.*" | head -1

Record: which IP serves now, and that a user journey works. You need the "before" to prove the "after."

3. Start a continuous probe (your measurement instrument)

In a separate terminal, hit the app once a second and log which region answers (have each region's app return a header or body identifying itself, or resolve+log the IP):

while true; do
  echo "$(date +%T) $(curl -s -m 2 -o /dev/null -w '%{http_code}' http://$FQDN/) $(dig +short $FQDN | head -1)"
  sleep 1
done | tee drill-log.txt

This log is how you'll measure t_cutover and the DNS-TTL tail. Leave it running through the drill.

4. Execute the failover (trigger + data + cutover)

Note the wall-clock time, then run your runbook's steps:

date +%T   # T0 — declare
# (data promotion step for your data store — e.g. az storage account failover, if applicable)
az network traffic-manager endpoint update -g dr-rg --profile-name app-tm \
  -n primary --type externalEndpoints --endpoint-status Disabled
date +%T   # cutover initiated

Watch drill-log.txt: note the timestamp when the answered IP first becomes the secondary, and when it stops ever being the primary.

5. Scale up the standby (if pilot light / warm standby)

az aks nodepool scale -g dr-secondary-rg --cluster-name aks-wus -n nodepool1 --node-count 3
kubectl --context aks-wus scale deployment web --replicas=3
kubectl --context aks-wus get pods -w

Time this phase — it's t_scale_up, and it's often the biggest surprise (node provisioning + image pull + readiness).

6. Verify like a user, not like a dashboard

Confirm a real end-to-end journey in the secondary — not just "pods Running":

curl -s "http://$FQDN/" | grep -o "Welcome.*"      # served from secondary now
# hit an endpoint that touches the data store, and confirm the data is present & current

Record achieved RTO (T0 → first successful secondary response for a new client) and, if you promoted a replicated data store, the RPO (how far behind the secondary's data was).

7. Diagnose-and-fix: the failover that didn't redirect traffic (DNS TTL)

This is the module's core lesson. Look at drill-log.txt: you'll almost certainly see the answered IP flip to the secondary quickly, but requests that reused an already-resolved connection or a cached resolver answer keep landing on the (now disabled) primary for up to the TTL — appearing as a tail of failures or old-region responses after cutover. Reproduce it sharply:

# Set a deliberately long TTL, re-baseline, then fail over and watch the tail:
az network traffic-manager profile update -g dr-rg -n app-tm --ttl 300
sleep 5
# re-run steps 3-4 and observe: cutover at the TM level is instant, but clients
# keep hitting the old IP for up to ~300s.

Findings to record: Traffic Manager switched immediately, yet real failover (from the client's perspective) lagged by up to the TTL. Fixes: (1) lower the TTL before incidents (--ttl 30 or lower) so the cached window is small; you cannot fix it mid-incident because the long TTL is already cached. (2) For sub-TTL failover, move the entry point to Front Door (L7 edge failover, no client DNS in the path) — re-run the drill through the Front Door endpoint and confirm the tail is seconds, not minutes. Update dr-runbook.md to note the realistic client-side failover window and the TTL precondition.

8. Record results and fix the runbook

Write a short drill report: achieved RTO decomposed by phase (t_detect, t_promote, t_cutover, t_scale_up, t_verify), achieved RPO, and a numbered list of every runbook step that was wrong, missing, ambiguous, or slower than assumed. For each, note the fix. Then edit dr-runbook.md accordingly. This report is the deliverable — a drill with no recorded discrepancies means you didn't look hard enough.

9. Fail back, then clean up

Failback is a real step (and often harder). Re-enable the primary and shift back, being explicit about any data written to the secondary during the drill:

az network traffic-manager endpoint update -g dr-rg --profile-name app-tm \
  -n primary --type externalEndpoints --endpoint-status Enabled
az network traffic-manager profile update -g dr-rg -n app-tm --ttl 30   # keep it low
# Then tear the expensive standby down:
az group delete -n dr-secondary-rg --yes --no-wait
az afd profile delete -g dr-rg --profile-name fd --yes 2>/dev/null
az network traffic-manager profile delete -g dr-rg -n app-tm
az group delete -n dr-rg --yes --no-wait
az aks list -o table   # confirm the DR cluster is gone

Expected: back on the primary, and the billable second region destroyed. Never leave a drilled-up standby running — that's the doubled bill.

Independent challenge

Take a stateful system (the AKS + database environment from track 07 or track 14's capstone). Write and then actually execute a regional-failover drill for it that includes a data promotion step, not just traffic — promote a geo-replicated data store (Azure SQL failover group, storage account failover, or a promoted read replica) and measure the RPO you really achieved (how much data the async replication gap cost you), alongside the decomposed RTO. Record a drill report and fix at least three concrete runbook defects you find. Reference module 00 (your RTO/RPO targets), module 01 (the substrate), track 20 / module 05 (declare it as an incident with roles), and track 14 (the data layer). Destroy the second region and any promoted-back resources immediately after — this drill runs real duplicated infrastructure.

Common mistakes & troubleshooting

• Writing the plan and never running it. An unexecuted runbook is a hypothesis; the first drill always finds broken steps. The drill, with a recorded report, is the actual deliverable.
• Forgetting the DNS-TTL term in the failover time. Traffic Manager switching is instant; real client failover lags by the cached TTL. Lower TTL before incidents or use Front Door — you can't fix it mid-outage.
• Verifying pods, not user journeys. "Pods Running in the secondary" isn't recovery; a real end-to-end request touching the data layer is. Bake user-journey verification into the runbook.
• Ignoring failback. Returning to the primary is often harder than leaving it — you must reconcile data written to the secondary. A plan with no failback strategy strands you in the DR region.
• Runbook depends on one person or un-scripted tribal knowledge. Use named roles (track 20 / module 05) and exact commands, so anyone on-call can execute it.
• Cost pitfall — leaving the drilled-up standby running (ties to track 21). Scaling the secondary to full capacity for a drill and forgetting to scale it back (or destroy it) is a classic way to double your bill silently for weeks. Make "scale back / destroy the standby" the final, non-optional runbook step, and confirm with az aks list that it's gone.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. Who is the audience a DR runbook is written for, and how does that shape its form?
2. List the seven phases of a regional-failover runbook in order.
3. Decompose achieved RTO into its component terms, and say which term the DNS TTL lives in and which router avoids that term entirely.
4. Why is "the plan is written" not the same as "the plan works," and what is the actual deliverable of this module?
5. In the drill, Traffic Manager switched to the secondary instantly, yet users kept hitting the dead primary. Explain why, and give the fix that must be applied before the incident.
6. Why is verifying "pods are Running in the secondary" insufficient, and what should you verify instead?
7. Why is failback often harder than the failover itself?
8. Which track/module's incident-response roles does the runbook reference, and why use named roles rather than named people?

Cumulative review

Closed-book. Cover the answers and write each out first — this mixes modules 00-02, roughly the first third of the track.

1. Define RTO and RPO, and name the DR strategy that minimizes each at the highest cost.
2. Name the four DR strategies in cost order and give the RTO/RPO shape of each.
3. Which failure domain does multi-region DR not cover, and what layer must you add to cover it?
4. Why does choosing Azure's paired region as your DR region help, and name two things it gives you for free.
5. Explain the core Traffic-Manager-vs-Front-Door difference and why it determines failover speed.
6. Decompose achieved RTO into its phases, and say which phase the DNS TTL lives in and which router avoids that term.
7. In the drill, "Traffic Manager switched instantly but users kept hitting the dead region." Explain why, and give the fix that must be applied before the incident.
8. Why is a DR runbook written in numbered copy-pasteable steps with per-step verification, rather than prose, and whose incident-response roles (which track/module) does it reference?
9. What makes failback often harder than failover?
10. Give the one FinOps rule that keeps DR drills from doubling your bill, tying it to track 21.

Next

03-platform-level-backup-strategy — you can fail over a running system to another region. But some disasters (corruption, a botched cluster upgrade, "delete the whole resource group") need backups, not failover. Time to extend track 14's database backups to the whole platform: VMs, disks, the cluster itself, and Terraform as a recovery mechanism.