00 - DR Concepts: RTO, RPO, and the Strategy Spectrum

Why this matters

Disaster recovery is the one area where teams routinely spend either far too little (a nightly backup nobody has ever restored) or far too much (a full active-active clone of production for a system that could tolerate an hour of downtime). Both mistakes come from skipping the only step that matters first: writing down, as numbers, how much downtime and how much data loss the business can actually absorb. This module gives you those two numbers — RTO and RPO — and the spectrum of architectures that trade cost against them, so every design decision in the rest of the track is anchored to a target instead of a vibe.

Concepts

RTO and RPO, defined precisely

Track 14 / module 04 introduced these for a single database; here they scale up to a whole system, and the precision matters more:

• RTO — Recovery Time Objective: the maximum acceptable wall-clock time between a disaster starting and service being restored. It's a duration you commit to, measured from "the region is down" to "customers can use the system again." An RTO of 4 hours means a 4-hour outage is within tolerance; a 6-hour one is a breach.
• RPO — Recovery Point Objective: the maximum acceptable amount of data, measured in time, that you can afford to lose. An RPO of 5 minutes means that after recovery, at most the last 5 minutes of writes may be gone. RPO is set by how continuously you replicate data, not by how fast you recover.

The trap is treating them as one number. They're independent: you can have a tiny RPO (data replicated to the second) but a large RTO (it takes hours to stand the application back up), or vice versa. Every DR strategy is really a point on a 2-D (RTO, RPO) plane, and the whole design job is hitting a target point at the lowest cost.

How to actually choose RTO and RPO

You don't pick these; the business does, and your job is to force the conversation with concrete numbers. A workable method:

1. Per system, not globally. The checkout service and the internal analytics dashboard have wildly different tolerances. Set RTO/RPO per service tier (e.g. Tier 0 = revenue-critical, Tier 2 = internal tooling).
2. Translate downtime and data loss into money and trust. "What does one hour down cost us in lost revenue, SLA credits, and reputation? What does losing the last 30 minutes of orders cost?" A number the business winces at is a real target; "as fast as possible" is not.
3. Reconcile with the SLO. This is the direct tie to track 20: your availability SLO already implies a downtime budget. A DR RTO that would blow the entire quarter's error budget in a single event is a mismatch — either the RTO is too loose or the SLO is unfundable.
4. Cost the strategy that meets it, then sanity-check. Once you know the cheapest strategy (next section) that hits the target, put its monthly cost in front of the business. Often the target tightens or loosens once the price of meeting it is visible — which is exactly the FinOps conversation from track 21.

The output is a small table: system, tier, RTO, RPO, chosen strategy, monthly cost. That table is the DR strategy.

The DR strategy spectrum

There are four canonical strategies, ordered from cheapest/slowest to most-expensive/fastest. Each is a different point on the (RTO, RPO) plane:

• Backup and restore. Data is backed up to durable, ideally geo-redundant storage; on disaster you provision fresh infrastructure and restore. Cheapest — you pay only for backup storage — but slowest: RTO of hours to a day (you rebuild everything), RPO tied to backup frequency. This is track 14 / module 04's world, extended to the whole system.
• Pilot light. A minimal always-on core in the second region — the database replicating continuously, critical config present — but the application tier is off or scaled to zero. On disaster you "turn up the lights": scale out the app tier and cut traffic over. Low steady cost (you pay for replication + a tiny footprint), moderate RTO (minutes to an hour to scale up), small RPO (continuous replication).
• Warm standby. A scaled-down but running copy of the full system in the second region, continuously handling health checks (and optionally a little real traffic). On disaster you scale it up and shift traffic. Higher steady cost (a second running environment), fast RTO (minutes), small RPO.
• Active-active (hot standby / multi-site). Full production capacity running in both regions simultaneously, both serving live traffic behind a global router. On disaster you simply stop routing to the dead region — near-zero RTO, near-zero RPO. Most expensive (you run 2× everything, plus the hard problem of multi-region data consistency) and most complex.

 cost / complexity  ──────────────────────────────────►
 backup&restore   pilot light   warm standby   active-active
 RTO: hours→day    RTO: ~1hr     RTO: minutes   RTO: ~seconds
 RPO: hrs (backup) RPO: seconds  RPO: seconds   RPO: ~zero
 $                 $$            $$$            $$$$

Cost and complexity are the real axis

The reason there's a spectrum and not just "always do active-active" is that each step right roughly multiplies both cost and operational complexity. Active-active doesn't only cost 2× the infrastructure — it forces you to solve bidirectional data consistency (two regions both writing to the same logical dataset), conflict resolution, and split-brain avoidance, which is genuinely hard engineering. Warm standby avoids the consistency nightmare (one region is authoritative) but still runs a second environment. Pilot light shrinks that second environment to almost nothing. The discipline is to pick the leftmost (cheapest) strategy that still meets the RTO/RPO the business signed off on — anything further right is money and complexity spent buying reliability nobody asked for. This is the same right-sizing instinct as track 21's FinOps, applied to resilience.

Failure domains: what you're actually recovering from

"Disaster" is vague; design against specific failure domains, because they demand different strategies:

• Single resource failure (a VM, a pod, a disk) — handled by in-region redundancy (replicas, availability zones), not DR. Chaos engineering (module 04+) targets this layer.
• Availability Zone failure — one datacenter within a region. Mitigated by zone-redundant deployments inside one region (cheaper than a second region).
• Regional failure — the entire Azure region is unreachable. This is what multi-region DR (modules 01-02) exists for.
• Data corruption / human error / ransomware — logical damage that replicates to every region instantly. Multi-region does not protect you here; only point-in-time backups do (track 14 / module 04). A complete DR plan needs both a regional strategy and a backup strategy, because they cover different failure domains.

Command reference

This module is concept-heavy; the commands are for inspecting the building blocks (regions, zones, replication options) you'll design against. Flag-by-flag breakdowns are given for the multi-flag examples.

Flag breakdown — listing paired regions (Azure pairs each region with another in the same geography for platform-level geo-replication):

az account list-locations \
  --query "[?metadata.pairedRegion!=null].{Region:name, PairedWith:metadata.pairedRegion[0].name}" \
  -o table

• --query "[?metadata.pairedRegion!=null]" — JMESPath filter: only regions that have a pair.
• .{Region:name, PairedWith:...} — reshape each result into two named columns.
• metadata.pairedRegion[0].name — the paired region's name (e.g. eastus pairs with westus). GRS storage replicates to this region automatically; choosing your DR region to match the pair means storage geo-replication is free and built in.
• -o table — human-readable output.

Flag breakdown — checking Availability Zone support before choosing zone-redundant vs. multi-region:

az vm list-skus -l eastus --zone --query "[?resourceType=='virtualMachines'].name" -o tsv | sort -u | head

• -l eastus — the region to check.
• --zone — restrict to SKUs that support zones.
• --query "[?resourceType=='virtualMachines'].name" — just the VM SKU names.
• -o tsv | sort -u — dedupe. If your workload's SKU is here, a zone-redundant single-region design may meet your RTO/RPO far cheaper than a second region.

Hands-on exercises

No expensive resources are created here — this module is about turning fuzzy requirements into concrete targets and inspecting the substrate. Do the writing exercises for real; the numbers you produce feed every later module and the capstone.

1. Inventory your regions and their pairs

az account list-locations -o table
az account list-locations \
  --query "[?metadata.pairedRegion!=null].{Region:name, PairedWith:metadata.pairedRegion[0].name}" \
  -o table

Find the pair for the region you've used all curriculum (likely eastus). Write down: your primary region, its Azure pair, and one non-paired region that's also geographically distant. You'll choose a DR region in module 01; this is the shortlist.

2. Read the RPO already baked into your storage

# Create a throwaway storage account to inspect (or reuse an existing one):
az group create -n dr-concepts-rg -l eastus
az storage account create -n drconcepts$RANDOM -g dr-concepts-rg -l eastus --sku Standard_GRS
az storage account list -g dr-concepts-rg --query "[].{name:name, sku:sku.name}" -o table

Standard_GRS means data is asynchronously copied to the paired region. Note that "asynchronously" is an RPO: GRS has a documented RPO of typically well under 15 minutes but not zero — a regional disaster can lose the last few minutes of writes that hadn't replicated yet. Write that down as the RPO floor for anything stored this way.

3. Build your first RTO/RPO table

For a hypothetical (or real) system with three services — a customer-facing API, its database, and an internal reporting dashboard — write a table:

Fill in defensible numbers and a one-line justification each. There's no answer key — the skill is producing specific numbers you could defend to a business owner, not "as fast as possible." Keep this table; you'll refine it in the capstone.

4. Map each service to a strategy

Extend your table with two columns: chosen strategy (backup/restore, pilot light, warm standby, or active-active) and rough monthly cost multiplier vs. single-region (1×, ~1.2×, ~1.5×, 2×+). Force yourself to justify why the reporting dashboard is not active-active — that's the right-sizing muscle.

5. Reconcile a strategy against an SLO (track 20 tie-in)

Suppose the customer API's SLO is 99.9% monthly availability. That's a downtime budget of ~43 minutes per month. Answer in writing: does a backup-and-restore strategy with an RTO of 6 hours fit inside that budget? (No — a single DR event blows ~8× the entire month's budget.) What's the loosest RTO that a single regional event could consume without breaching 99.9% for the month, and what strategy does that push you toward?

6. Separate the failure domains

For each of these events, write which mitigation actually helps — in-region redundancy, multi-region DR, or point-in-time backups — and note that some need more than one:

1. A single pod OOM-kills.
2. An entire Azure region goes offline for 3 hours.
3. A bad migration drops a table at 10:00, noticed at 10:20.
4. An Availability Zone loses power.

The point: multi-region DR does nothing for #3 (the drop replicates instantly), and backups do nothing for #2's downtime (only its data). A real plan needs both layers. This is the exact split you'll design in modules 02-03.

7. Diagnose-and-fix: the "we have DR" claim that doesn't

A team says "we're covered — everything's in GRS storage, so we have disaster recovery." Interrogate the claim in writing and find the holes:

• GRS covers data durability across regions, at an RPO of minutes — but what's the RTO? (There is none: GRS gives you data in the other region, not a running application. Someone still has to stand up the app tier, DNS, and networking — that could be many hours.)
• Can they even read the geo-replicated copy on demand, or is it Standard_GRS (no read access until Microsoft initiates failover) rather than Standard_RA-GRS (read-access geo-redundant)?
• Does GRS protect against the dropped table in #3 above? (No — corruption replicates.)

Write the corrected statement: what GRS actually provides (a data RPO), and the three things still missing before this is "disaster recovery" (a defined RTO, a way to stand up compute in the second region, and a point-in-time backup layer for logical damage). This is the module's core lesson — "we have replication" is not "we have DR."

8. Clean up

az group delete -n dr-concepts-rg --yes --no-wait

Expected: the throwaway storage account and its group are gone. Nothing here was expensive, but the habit — every exercise ends by deleting what it created — is the one that keeps this track from generating a real bill in later modules.

Independent challenge

Pick one real system you understand well (something from an earlier track's capstone is ideal — the AKS + ACR + database environment from track 07's or track 14's capstone, or the multi-module Terraform environment from track 09's capstone). Draft a one-page DR strategy document for it without any template: identify each distinct component, assign each a tier and defensible RTO/RPO, choose the leftmost strategy on the spectrum that meets each target, and — critically — write a short paragraph justifying why you did not choose a more expensive strategy for at least one component, reconciling your RTO against a plausible SLO for it (drawing on track 20). End with a rough monthly-cost sanity check versus single-region (the track 21 lens). No resources need be created — this is the design artifact that the capstone will make you actually build and test.

Common mistakes & troubleshooting

• Collapsing RTO and RPO into one number. They're independent axes — fast recovery (RTO) and little data loss (RPO) are bought by different mechanisms (compute/automation vs. replication frequency). Always state both.
• "We have backups" or "we have GRS" = "we have DR." Replication and backups give data-side guarantees (RPO); neither gives you a running application in the other region on a deadline (RTO). DR is data plus compute plus traffic plus a tested procedure.
• Designing DR globally instead of per-tier. One system rarely has one tolerance. A blanket "everything active-active" is the most common way to spend 4× on components that could tolerate a 6-hour restore.
• Ignoring the logical-damage failure domain. Multi-region replicates corruption instantly; only point-in-time backups (track 14 / module 04) recover from a bad migration or DELETE. A DR plan without a backup layer is incomplete no matter how many regions it spans.
• Cost pitfall — reflexively reaching for active-active (ties to track 21). Active-active is not "2× cost," it's often more: 2× compute, cross-region egress charges, geo-replicated storage, a global router, plus the engineering cost of solving bidirectional data consistency. For the large majority of systems, warm standby or pilot light meets the real RTO/RPO at a fraction of the price. The FinOps question — "what is the cheapest strategy that meets the signed-off target?" — is the correct default, not "what's the most resilient thing we can build?"
• Setting targets without the business. RTO/RPO are business decisions informed by engineering cost. Numbers an engineer invents alone tend to be either heroically tight (expensive) or quietly loose (a nasty surprise during a real outage).

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. Define RTO and RPO, and give one mechanism that primarily improves each.
2. Why can't RTO and RPO be a single number? Give an example of small-RPO but large-RTO.
3. Name the four strategies on the DR spectrum in order of increasing cost, with a one-line RTO/RPO characterization of each.
4. A system's SLO is 99.9% monthly availability. Roughly what downtime budget is that per month, and why does a 6-hour-RTO backup-and-restore strategy conflict with it?
5. Your data is in Standard_GRS storage. What DR guarantee does that give you, and what does it explicitly not give you?
6. Which failure domain does multi-region DR not protect against, and what does?
7. Give the one-sentence rule for choosing a position on the strategy spectrum, and tie it to the FinOps discipline from track 21.

Next

01-multi-region-architecture-on-azure — you have targets and a chosen strategy on paper. Now build the substrate that makes any of them possible: a second Azure region, geo-redundant data, and a global traffic router that can send users to whichever region is alive.