Module 11: Capstone Project

Why this matters

Every module in this track taught one dimension of operational readiness in isolation: handle errors deliberately, manage config safely, log as a structured stream, aggregate those logs centrally, emit metrics, trace requests across services, alert on symptoms without fatigue, shut down gracefully, and structure the whole thing as a 12-factor app. None of these is worth much alone. A service with beautiful metrics but swallowed exceptions is blind to its own failures; one with perfect logging but no graceful shutdown sheds errors on every deploy; one that's observable but un-alertable tells you nothing until a user complains. The gap between "my service works" and "my service is operationally ready" is exactly the gap between having these capabilities individually and having them work together, under real failure conditions.

This capstone makes you close that gap on a single real service. The point isn't to build a large application — it's to take a modest FastAPI service and make it genuinely production-ready across every dimension at once, then prove each property by breaking things, not by demonstrating a happy path. Anyone can show a dashboard when everything is green. Operational readiness is what you have when a dependency dies, a deploy rolls out mid-request, an error rate climbs, and a container gets SIGKILL'd — and your service handles each one visibly, safely, and without waking anyone unnecessarily. If you can build this and defend every choice, you've absorbed the track. If you get stuck, the module that taught the piece is named throughout — go back and redo its exercises rather than guessing.

The project

Build (or take an existing) FastAPI service — an "orders" or "documents" API is plenty — and make it operationally ready end to end. The domain is deliberately small so all your effort goes into the operational qualities, not the features. It must depend on at least one backing service (a database) and call at least one flaky external dependency (a real or simulated third-party API — e.g. a payment or notification provider) so that failure, degradation, retries, and tracing across a boundary all have something real to act on. Run it, and its backing services, locally in Docker.

The deliverable is the service plus a written "operational readiness report" that walks through each acceptance item with evidence (screenshots, log excerpts, dashboard panels, PromQL, trace waterfalls) — and, crucially, evidence from the failure demonstrations, not just the happy path.

Acceptance checklist

Your build is done when all of these are true and you can demonstrate each — most by deliberately causing a failure and showing the correct behaviour.

Error handling (modules 00-01)

The flaky external dependency is called with timeouts + retries with backoff for transient failures, and either a graceful fallback/degraded response or a translated, propagated error when it truly fails — never a swallowed exception.
A global exception handler returns safe, correctly-shaped error responses (right status code, no leaked internals/stack traces) that include a correlation id the client can quote.

Config (modules 02-03)

All config comes from the environment via typed Settings; secrets are SecretStr; the repo passes the "could I open-source this right now?" test (no hardcoded credentials).
Log level and environment are config-driven, and flipping the service to DEBUG requires no code change or rebuild.

Logging (modules 04-05)

All logs are structured JSON to stdout, at deliberate levels (no crying wolf), with a correlation id bound per request and propagated to the external dependency; no secret or PII ever appears (a redaction processor proves it).
Logs are shipped to a central store (Loki/ELK) and a single request_id/trace_id query reconstructs one request's full story.

Metrics (module 06)

The service exposes /metrics with the full RED set (rate, errors, duration histogram) plus an in-flight gauge, all with low-cardinality labels (no user_id/request_id as labels), scraped by Prometheus.

Tracing (module 07)

Requests are traced with OpenTelemetry, the trace propagates across the call to the external dependency (one distributed trace, viewable as a waterfall), and log lines carry the trace_id so you can pivot metric → trace → logs.

Alerting (module 08)

There are SLIs/SLOs (availability + latency) with an error budget, and multi-window burn-rate alerts that page only on a sustained fast burn and ticket on a slow burn — and nothing non-actionable pages.
At least three cause signals (CPU, restarts, pool saturation) are on a dashboard/ticket, explicitly not paging, with justification.

Graceful shutdown (module 09)

On SIGTERM the service drains in-flight requests, flips readiness to not-ready (while liveness keeps passing), cancels/awaits background tasks idempotently, and closes pools — all within the grace period.
Distinct liveness (intrinsic) and readiness probes exist; liveness does not check dependencies.

12-factor (module 10)

The service is stateless (no in-memory/local-disk persistent state — it runs correctly as multiple replicas behind a load balancer), built once as an immutable image and released with per-environment config.

Failure demonstrations (the heart of the capstone)

A green happy-path proves almost nothing this track cared about. Your report must demonstrate the service behaving correctly under each of these induced failures:

Kill the external dependency. Show the request either degrades gracefully or fails with a safe 5xx + correlation id; the failure is logged at the right level with a traceback; the error-budget burn rises but a brief outage does not page (only a sustained one does).
Spike the error rate deliberately. Show the metric dashboard catch it (that/when), a trace of a failed request pinpoint the failing hop (where), and the logs for that trace_id explain it (why) — the full three-pillar loop on your own data.
Roll a deploy / docker stop under load. With several requests and a background task in flight, show zero dropped requests, the in-flight gauge return to zero, work neither lost nor double-applied, and the SLO alert not fire on the routine deploy.
SIGKILL vs SIGTERM. Contrast a hard kill (dropped requests, no cleanup) with a graceful SIGTERM (drained, cleaned up) to prove your shutdown path actually runs.
Run two/three replicas. Show a stateful operation (a session, a counter, an upload) behaves identically regardless of which replica serves it — proving statelessness — and that it would have broken with in-memory state.
Try to leak a secret and PII. Show a deliberate log.info(**body) gets redacted and that SecretStr renders masked.

Hints (design, not code)

Let one correlation id be the spine of everything

The single request_id/trace_id — generated at the edge, bound to the log context, propagated on the outbound call, stamped on every log line, and carried as the trace id — is what ties all eight dimensions into one system. It's how a metric spike leads to a trace leads to the exact logs; how a client's error response leads back to the server-side traceback; how one request's story survives across services and central storage. If you build nothing else first, build the correlation-id backbone (module 05 + 07), because every other property plugs into it. When a piece feels disconnected, the fix is almost always "make it carry / read the shared id."

Where each property has to live

Map each capability to its natural home so they compose instead of tangle: error handling and correlation-id binding in middleware + a global handler; config in a Settings object injected at startup; logging setup + redaction in one configure_logging call; metrics in middleware (RED) + a few business counters; tracing via auto-instrumentation (FastAPI + the HTTP client) so spans and propagation are mostly free; alerting as PromQL rules + Alertmanager routing (config, not app code); graceful shutdown in the lifespan block + probe endpoints; statelessness by pushing every piece of state into a backing service. Notice most of this is cross-cutting (middleware, lifespan, config) — operational readiness is largely built around your handlers, not inside them, which is why a small domain suffices.

Design the alerts backward from the SLO

Don't start from "what can I measure?" — start from "what promise am I making to users?" Pick an availability SLO (e.g. 99.9%) and a latency SLO (e.g. 99% under 300ms), derive the error budgets, and let every paging decision fall out of "are we burning that budget too fast?" (the multi-window burn-rate rule). Then the cause signals (CPU, restarts, pool saturation) obviously become dashboard/ticket items, because none of them directly means "the promise to users is breaking." If you find yourself wanting to page on a cause, ask the litmus test — user hurt? human must act now? — and it'll almost always demote itself.

Prove it by breaking it — script the failures

The failure demonstrations are the capstone, so make them repeatable: a small script or set of commands that kills the dependency, injects errors, docker stops under load, sends SIGKILL vs SIGTERM, and scales replicas — each paired with the evidence it produces. Idempotency is what lets several of these be safe (cancel a background task mid-flight; retry a failed external call) — the same property from module 00 that made retries safe makes interruption and redelivery safe. A demonstration that only ever shows green has skipped the entire subject: readiness is defined by behaviour under failure.