07 - Resilience Patterns Review and Gap Analysis

Why this matters

Modules 04-06 found resilience gaps by breaking things. This module finds them by reading the system — a structured audit of the resilience patterns that should be present (timeouts, retries, circuit breakers from track 13, health probes, graceful degradation, redundancy) and where each is missing, wrong, or untested. Chaos experiments prove specific gaps exist; a gap analysis maps the whole surface and prioritizes the fixes so limited engineering time goes to the failures that would actually hurt. It's the bridge from "we ran some experiments" to "here is the ranked list of what to fix and why" — the artifact a real team acts on, and the one your capstone produces.

Concepts

The resilience patterns to audit

You've met these across the curriculum; a gap analysis checks each is actually present and correct at every relevant call:

• Timeouts — every network call (service→service, service→database, service→external API) must have a bounded timeout. The default in most HTTP clients is no timeout or a very long one — the single most common resilience bug, and the one module 05's latency experiment exposes: without a timeout, a slow dependency hangs the caller indefinitely.
• Retries (with backoff and jitter) — transient failures should be retried, but bounded, with exponential backoff and jitter, and only for idempotent operations. Naive retries (immediate, unlimited, on everything) turn a blip into a self-inflicted DDoS — the "retry storm."
• Circuit breakers — when a dependency is clearly failing, stop calling it (open the circuit), fail fast, and periodically test if it's recovered. This is what prevents the cascade module 05's network experiment triggers. Track 13's service mesh provides these declaratively.
• Bulkheads — isolate resources (connection pools, thread pools) per dependency so one saturated dependency can't consume all of them and starve the rest — the isolation module 05's resource-exhaustion experiment tests.
• Health probes — liveness (restart if wedged), readiness (don't route to me until I can serve), and startup probes (track 03/07). Wrong probes cause both false failovers and traffic sent to dead pods (module 01's exercise 8).
• Graceful degradation — when a non-critical dependency is down, serve a reduced experience (cached data, a default, a disabled feature) rather than failing the whole request. The difference between "recommendations are temporarily unavailable" and a blank error page.
• Redundancy across fault domains — replicas spread across nodes/zones, PodDisruptionBudgets, no single-instance dependencies (module 04/05's SPOF findings).

Where these come from in the curriculum

The gap analysis is a synthesis, not new material — it's auditing patterns you already have the tools for:

• Track 13 (service mesh) — timeouts, retries, circuit breakers, outlier-detection as mesh config (Istio DestinationRule/VirtualService). A mesh makes these auditable in one place rather than scattered across application code.
• Track 03/07 (Kubernetes/AKS) — probes, resource requests/limits, topology spread, PodDisruptionBudgets.
• Track 12 (observability) — you can't audit what you can't see; tracing reveals every call that needs a timeout, and metrics reveal retry storms and saturation.
• Track 20 (SRE) — the SLOs tell you which gaps matter (a gap on a path that never affects an SLO is low priority), and error-budget burn during chaos tells you which are actively costing reliability.
• This track (00-06) — module 00's RTO/RPO targets and modules 04-06's experiment findings feed directly into what to audit and how to rank it.

The gap analysis method

A repeatable process, not a vibe check:

1. Map the dependency graph. List every component and every call between them (distributed tracing, track 12, is the fastest way — it is the call graph). You can't audit calls you don't know exist.
2. For each call/component, check each pattern. Does this call have a timeout? A bounded retry? A circuit breaker? Is this component redundant across fault domains? Does it degrade gracefully if its dependency is down? Produce a matrix: rows = calls/components, columns = patterns, cells = present/missing/wrong.
3. Identify single points of failure. Anything whose failure takes down a user-facing path with no fallback — the SPOFs modules 04-05 surface.
4. Assess each gap: likelihood × impact. How likely is this failure, and how bad if it happens (how many users, which SLO, how much revenue)? This is a risk assessment, borrowing the same likelihood×impact framing as threat modeling (track 11) and FinOps prioritization (track 21).
5. Prioritize and estimate. Rank gaps by risk, weigh against fix effort, and produce a ranked backlog: what to fix first, why, and roughly how much work. The output is a decision-ready document, not a wall of findings.

Prioritization: not every gap is worth fixing

The discipline that makes a gap analysis useful is ruthless prioritization — the same instinct as track 21's cost prioritization and track 20's error budget. A missing timeout on the checkout→payment call (high traffic, revenue path, tight SLO) is a P0; a missing circuit breaker on an internal admin-dashboard→metrics call that's already behind a degraded-mode fallback is a P3 you may never do. Rank by risk to an actual SLO / user journey, not by "which pattern is missing" in the abstract. A gap analysis that flags 200 issues with no ranking is as useless as none — the value is the top 5 that actually matter, with a defensible reason each.

Validate fixes the way you found the gaps

A gap analysis proposes fixes; chaos engineering verifies them. The loop closes: for each high-priority gap you fix (add a timeout, a circuit breaker, a zone spread), you re-run the chaos experiment that would have exposed it (module 05) and confirm the system now degrades gracefully. An unverified fix is just another untested assumption — the exact trap this whole track exists to break. This is why the capstone requires both the gap analysis and chaos experiments: the analysis says what to fix, the experiments prove you fixed it.

Command reference

The "commands" here are inspection and audit tools, plus the mesh/Kubernetes config where the patterns live.

Audit snippet — check a Deployment has readiness and liveness probes and that they differ sensibly (a liveness probe that's really a readiness check causes restart loops):

kubectl get deploy web -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\n  live: "}{.livenessProbe}{"\n  ready: "}{.readinessProbe}{"\n"}{end}'

• Confirms both exist. Missing readiness → traffic to not-yet-ready pods (failed requests on rollout). Missing liveness → a wedged pod never restarts. Liveness == readiness on a slow dependency → cascading restarts when the dependency is slow (the probe fails, the pod restarts, worsening load). The audit flags all three.

Audit snippet — an Istio VirtualService/DestinationRule resilience check (track 13):

kubectl get virtualservice web -o yaml | grep -A4 -iE "timeout|retries"
kubectl get destinationrule web -o yaml | grep -A6 -iE "outlierDetection|connectionPool"

• timeout on the route — present and shorter than upstream timeouts? A call with no timeout inherits the mesh/app default (often too long or none).
• retries — bounded attempts with a perTryTimeout, not unlimited.
• outlierDetection — the circuit breaker: ejects failing hosts. Absent = no circuit breaking, so a bad instance keeps taking traffic.
• connectionPool — the bulkhead: caps concurrent connections/requests so one dependency can't exhaust the pool.

Hands-on exercises

Audit a real system you've built (a track 07/13/14 environment with the track-12 stack). No new billable multi-region infra is required — this is mostly reading and small fixes, verified with module-05 chaos. Clean up any chaos objects and dedicated clusters in exercise 8.

1. Map the dependency graph from traces

Use distributed tracing (track 12) under load to enumerate every call: which services call which, and which external dependencies/databases each touches. Draw it (even on paper). Every edge is a call that needs a timeout and possibly a circuit breaker — you can't audit edges you didn't know existed, and tracing usually reveals a few surprises.

2. Build the resilience matrix

Make a table: rows = each call/component from exercise 1, columns = timeout, bounded-retry, circuit-breaker, redundant-across-fault-domains, graceful-degradation. Fill each cell present / missing / wrong by inspecting the mesh config, Deployments, and code. This matrix is the core artifact.

3. Audit health probes

kubectl get deploy -A -o json | jq -r '.items[] | .metadata.name + ": live=" + (.spec.template.spec.containers[0].livenessProbe!=null|tostring) + " ready=" + (.spec.template.spec.containers[0].readinessProbe!=null|tostring)'

Flag every workload missing a readiness probe (traffic to unready pods) or missing liveness (wedged pods never restart), and any liveness probe that would fail under dependency slowness (restart-loop risk). Add these to the matrix.

4. Audit timeouts, retries, and circuit breakers (track 13)

kubectl get virtualservice,destinationrule -A -o yaml | grep -iE "timeout|retries|outlierDetection|connectionPool" || echo "NONE FOUND — every call is unbounded"

The common, alarming finding: no timeouts or circuit breakers configured anywhere — meaning every inter-service call is unbounded and one slow dependency can hang everything (module 05's cascade). Record which calls lack each pattern.

5. Hunt single points of failure

kubectl get pods -A -o wide | awk '{print $2, $8}'    # pod, node — are "replicas" on one node?
kubectl get pdb -A                                     # any PodDisruptionBudgets?
# find single-instance backing services (a lone db/cache pod with no replica)

List every SPOF: replicas that don't span fault domains, single-instance dependencies, no PDBs. These are usually the highest-impact gaps (a whole user-facing path dies with no fallback).

6. Score and prioritize (the actual deliverable)

For each gap in the matrix, assign likelihood (how often could this fail?) and impact (which SLO/user journey, how many users, revenue?) — using your track-20 SLOs to decide impact. Rank by risk, then annotate rough fix effort. Produce a prioritized gap-analysis document: top gaps first, each with a one-line justification tying it to a real SLO/journey. Ruthlessly deprioritize gaps on paths no SLO cares about. This ranked doc — not the raw matrix — is what a team acts on and what the capstone requires.

7. Diagnose-and-fix: fix the top gap and verify it with chaos

Take your #1 gap — most likely a missing timeout + circuit breaker on a critical call. Fix it (add a mesh timeout + outlierDetection, or a client timeout), then re-run the module-05 experiment that would have exposed it and confirm the system now degrades gracefully instead of cascading:

# add timeout + circuit breaker (Istio example):
kubectl apply -f web-vs-with-timeout.yaml      # timeout: 2s, bounded retries
kubectl apply -f db-dr-with-outlier.yaml       # outlierDetection = circuit breaker
# now re-run the latency experiment from module 05:
kubectl apply -f networkchaos.yaml             # 5s latency on the dependency
# verify on Grafana: caller now times out at 2s and sheds load / serves degraded,
# instead of hanging until resource exhaustion.

The point: the fix is only real once the experiment that found the gap now passes. An unverified resilience fix is just another untested assumption — the whole track's thesis. Record before/after dashboards.

8. Clean up

kubectl delete podchaos,networkchaos,stresschaos --all -A 2>/dev/null
# remove any dedicated cluster/infra stood up for the audit:
az group delete -n <audit-rg> --yes --no-wait 2>/dev/null

Expected: no active chaos objects, no lingering dedicated infra. Keep the gap-analysis document — it's a deliverable, and the capstone builds on it.

Independent challenge

Produce a complete, prioritized resilience gap analysis for a real system (a track 07/13/14 capstone environment), then fix and verify the top two gaps. Map the dependency graph from track-12 traces, build the resilience matrix (timeouts / bounded retries / circuit breakers / cross-fault-domain redundancy / graceful degradation / probes), identify SPOFs, and rank every gap by likelihood×impact against your track-20 SLOs — producing a ranked document where each top item has a defensible one-line justification. Then remediate the two highest-priority gaps (e.g. add a timeout + circuit breaker via track 13's mesh, spread a single-node "replicated" workload across zones) and re-run the module-05 chaos experiment that would expose each, proving the fix holds. Draws on track 12 (call graph), track 13 (mesh patterns), track 20 (SLO-based prioritization), and modules 04-06 (the experiments that verify). Tear down any chaos objects or dedicated infra afterward.

Common mistakes & troubleshooting

• Auditing from memory instead of from traces. You'll miss exactly the calls that lack timeouts — the forgotten external dependency. Map the real call graph from track-12 tracing first.
• The missing-timeout blind spot. Most HTTP clients default to no (or a huge) timeout; an unbounded call hangs the caller when the dependency slows. This is the most common and most damaging single gap — check every call.
• Naive retries as a "fix." Immediate, unlimited retries on everything turn a blip into a retry storm and can take down the dependency you're retrying. Retries need backoff, jitter, bounds, and idempotency.
• Liveness probe doubling as a dependency check. A liveness probe that fails when a downstream is slow causes restart loops that worsen an incident. Liveness = "am I wedged"; readiness = "can I serve right now."
• A gap list with no prioritization. 200 unranked findings are as useless as none. Rank by likelihood×impact against real SLOs and present the top few that matter, each justified.
• Proposing fixes you never verify. A fix is a hypothesis until a chaos experiment (module 05) confirms the system now degrades gracefully. Unverified fixes are how "we hardened it" becomes the next outage.
• Cost pitfall — over-hardening low-value paths (ties to track 21). Redundancy, multi-zone spread, and active-active for a component that no SLO depends on spends real money (and complexity) for reliability nobody needs — the same over-provisioning FinOps guards against. Prioritize hardening spend by risk to actual SLOs, and be willing to leave low-impact gaps unfixed.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. Name five resilience patterns a gap analysis audits and the failure each guards against.
2. Why is a missing timeout the most commonly cited resilience bug, and what does it cause?
3. Why can naive retries make an incident worse, and what three properties make retries safe?
4. What's the difference between a liveness and a readiness probe, and what goes wrong if a liveness probe effectively checks a downstream dependency?
5. Give the five steps of the gap-analysis method.
6. Why must a gap analysis prioritize by likelihood×impact against SLOs rather than by "which pattern is missing"?
7. Why isn't a proposed fix "done" until you've run a chaos experiment against it?
8. Give an example of a resilience gap you'd deliberately not fix, and the FinOps reasoning (track 21).

Next

08-capstone-project — you now have every piece: RTO/RPO targets, a multi-region substrate, a tested DR runbook, platform backups, chaos discipline and tooling, the game-day practice, and a prioritization method. The capstone combines them into one end-to-end deliverable: a real DR plan, a Terraform failover you actually execute, chaos experiments you actually run, and a gap analysis you actually prioritize.