Module 18: Capstone Project

Roadmap

What this capstone is. Track 06 gave you the provider landscape one slice at a time: a first OpenAI call (Module 00), Chat Completions (Module 01), the Responses API and statefulness (Module 02), streaming (Module 03), model families and generations (Module 04), reasoning models (Module 05), function calling (Module 06), structured outputs and JSON Schema (Module 07), the embeddings API (Module 08), vision and audio inputs (Module 09), the Anthropic Claude API (Module 10), the Google Gemini API (Module 11), open-weight models (Module 12), the Hugging Face ecosystem (Module 13), small language models (Module 14), on-device and edge inference (Module 15), pricing, rate limits and quotas (Module 16), and provider abstraction and model routing (Module 17). This capstone asks you to put all of it behind one door: a multi-provider assistant gateway that your own applications call instead of calling providers directly.

It also leans on earlier tracks: token cost economics (Track 01, Module 12) and context windows (Track 01, Module 11) for the cost model, embedding models and embedding pipelines (Track 08, Module 02, Module 05), quantization and GGUF (Track 04, Module 15), and the memory and CPU arithmetic for the on-device route (Track 05, Module 05, Module 14).

There is no dataset, no solution, no code given and no single correct design. Two good gateways can draw their boundaries differently, choose different default policies and still both be right. What makes yours good is that every provider fact is sourced and dated, every failure mode has a test, every cost is accounted for, and the claim "my routing policy is better" rests on evidence you produced.

Afterwards you can build, test and defend the layer that sits between application code and model providers: normalising several APIs, surviving rate limits and outages, capping spend, routing by capability, cost and latency, and measuring whether a routing change helped. That layer exists in most real LLM systems, and it is where most production incidents (a surprise bill, a retry storm, an injected tool result) actually happen.

An honesty rule for this capstone. You may have no API keys. If you build local stand-ins, your engineering (parsing, streaming, tool loop, retries, caps, routing, evaluation) is genuinely exercised, but the model behind a stand-in is scripted, so no number from it says anything about any real model or provider. Label every result that came from a stand-in as such.

Concept

Visual diagram

A decomposition of the problem, not a prescribed design; your boundaries may differ.

   caller (CLI or local HTTP) sends ONE normalised request:
   messages, optional tools, optional schema, task tags, sensitivity flag, deadline, budget
                                          │
                                          ▼
 ┌─ GATEWAY (your code) ───────────────────────────────────────────────────────────────────┐
 │                                                                                         │
 │ ┌───────────┐   ┌────────────┐   ┌──────────────┐   ┌───────────────────────────────┐   │
 │ │  ROUTER   │──►│ SPEND GATE │──►│  RESILIENCE  │──►│           ADAPTERS            │   │
 │ │ policy on │   │ reserve the│   │ timeout,     │   │ normalise request, stream,    │   │
 │ │ capability│   │ worst case;│   │ retry, rate  │   │ tool calls, structured output,│   │
 │ │ cost,     │   │ refuse at  │   │ limiter,     │   │ embeddings, errors, usage     │   │
 │ │ latency,  │   │ the cap    │   │ fallback     │   │                               │   │
 │ │ sensitive │   │            │   │              │   │                               │   │
 │ └─────▲─────┘   └─────┬──────┘   └──────┬───────┘   └───────────────┬───────────────┘   │
 │       │               │ settle          │ every attempt             │ usage, status     │
 │       │               ▼                 ▼                           │                   │
 │ ┌─────┴─────────────────────────────────────────────────────────┐   │                   │
 │ │ LEDGER + LOGS + METRICS: route, reason, status, tokens, cost, │◄──┘                   │
 │ │ latency, provider request id; feeds router stats and caps     │                       │
 │ └───────────────────────────────────────────────────────────────┘                       │
 └─────────────────────────────────────────────────────────────────────────────────────────┘
          │ only adapters touch the network
          ▼
   ┌────────────────┐  ┌────────────────┐  ┌────────────────────────┐
   │ provider A     │  │ provider B     │  │ on-device / open-weight│
   │ real or        │  │ real or        │  │ route (local server,   │
   │ local stand-in │  │ local stand-in │  │ in-process, or stand-in│
   └────────────────┘  └────────────────┘  └────────────────────────┘

   Milestones, in build order (each ends with an offline test):

   M0 contracts ─► M1 adapters ─► M2 streaming ─► M3 tool loop ─► M4 schema ─► M5 embeddings ─┐
   ┌──────────────────────────────────────────────────────────────────────────────────────────┘
   └─► M6 retries, limits, fallback ─► M7 cost + cap ─► M8 routing ─► M9 evaluation ─► M10 report

   M9 is how you learn whether a change (a policy, a retry rule, a model id) helped; if it cannot
   tell two policies apart, the task set is too small. M10 also holds the security pass and the logs.

Project

The problem

A small product team ships four features that all need a language model: a support-ticket triager that must return JSON in a fixed shape, a chat helper that must stream, an order-assistant that calls internal tools (order lookup, unit conversion), and a help-centre search that needs embeddings. Today each feature calls one provider's SDK directly. Then a price changes, a provider has a bad afternoon, one feature gets rate-limited at peak, an experiment runs up an unexpected bill, and the privacy lead says tickets marked sensitive must never leave the company's machines.

You build the one internal service (a library plus a CLI, or a small local HTTP server; your choice) that all four features call instead.

Who uses it

Providers, keys and stand-ins

Requirements: at least two providers whose documented wire formats differ (two endpoints of the same API family count as one), plus at least one on-device or open-weight route. Real keys are optional.

If you have no keys, or want tests that never touch a network, build a local stand-in per provider dialect: a small HTTP server bound to 127.0.0.1 that implements the subset of that provider's documented shapes you use. A workable method:

1. Fetch the provider's official reference and guide pages yourself (starting points are under Further reading) and note the date you read them.
2. For each endpoint you use, write a shape sheet row: method and path, auth header, required request fields, the response fields you read (text, tool calls, usage, finish reason), streaming framing and event names, error envelope, rate-limit headers, request-id header. OpenAI also publishes a machine-readable specification in the openai/openai-openapi repository, which is a good cross-check for the OpenAI-shaped rows.
3. Make the scripted "model" deterministic and say so in the stand-in's name and in every log line it produces.
4. Add fault injection you can trigger per request: 429 with and without a retry hint, an overload or 5xx response, a slow response, a stream cut mid-way, an error event after a 200, malformed tool-call arguments, an unknown tool name, schema-invalid output, truncated output, and a 401.
5. Check the stand-in against the docs by re-reading them, not against your memory. If you can make even one live call to a real endpoint, diff the real response shape against your stand-in; the differences are findings.

What the official pages said when this module was written (checked Oct 2026; the docs are the source of truth and may have changed since, so re-read before you rely on any row):

Providers can document more than one way to do the same job (OpenAI has Chat Completions and the Responses API; the Gemini reference lists both an Interactions API and generateContent). Pick one interface per provider, record why, and do not assume a new one behaves like the old one.

Milestones

Each milestone ends with a test that runs against stand-ins only: offline, deterministic, no keys. "Uses" names the Track 06 modules to reread; the exact depth you need is your call.

M0. Contracts, configuration and the shape sheet. Uses: Modules 00, 01, 04, 16.

• Write the shape sheet (dated URLs). Define, in prose first, your normalised request, response, usage, finish reasons and error classes (retryable or not; auth, validation, quota, overload, timeout, refusal).
• Put routes, model identifiers, capability flags (tools, schema mode, streaming, embeddings, context and output limits) and prices in configuration, each with a source URL and a date. Nothing perishable in code.
• Done when: changing a model id, a price or a capability flag needs no code change, and a test fails loudly when a route is missing a field the router depends on.

M1. Normalised adapters, non-streaming. Uses: Modules 01, 02, 10, 11, 12, 17.

• One interface, one adapter per dialect plus one for the on-device route. Map system instructions, multi-turn history, output-length limits, stop reasons and usage. Keep the provider's request id and raw finish reason next to the normalised ones.
• Decide, and write down, what you refuse to normalise (features only one provider has) and how a caller reaches them: an escape hatch, or a typed capability error.
• If you use official SDKs, hide them behind your interface and account for their built-in retries (see M6).
• Done when: one caller test suite passes against every adapter, unknown extra response fields never crash parsing, and a "leaky abstraction log" lists each difference you hit and your decision.

M2. Streaming. Uses: Module 03, plus the streaming parts of Modules 10 and 11.

• A normalised event stream (text delta, tool-call fragment, usage, finish, error). Parse each provider's server-sent-event framing yourself, ignore event types you do not know, and reassemble tool-call arguments that arrive in fragments.
• Measure time to first token and total time. Propagate client disconnects upstream.
• Decide what a failure after output has started means for retry, fallback (never splice two models' text without saying so) and cost (partial usage still costs something).
• Done when: a stand-in stream cut mid-way yields a typed error with the partial text preserved and cost recorded, and an error event after a 200 is handled like any other failure.

M3. Tool-calling loop. Uses: Module 06, with each provider's tool shapes from Modules 01, 10 and 11.

• A tool registry (name, description, parameter schema, handler, timeout, read-only or side-effecting). A loop that calls the model, runs the requested tools (in parallel when several are requested), returns each result tied to the right call id, and repeats until a final answer or a limit.
• Validate arguments before running anything. Unknown tool, bad JSON, handler exception and timeout become results the model can see, or a hard stop, by an explicit policy.
• Treat tool output as untrusted data: size-limit it, keep it out of the instruction channel, allow-list tools per request, and gate side-effecting tools. Do not rely on the model to resist an injected instruction.
• Done when: the loop cannot exceed its step, time or spend budget, it works with two providers' tool shapes, and a stand-in scripted to obey an instruction planted in tool output still cannot cause a disallowed tool to run.

M4. Schema-validated structured output. Uses: Module 07; prompting side in Track 07: Prompt Engineering.

• The caller supplies a schema. Use a provider-enforced mode where the route's capability flags say one exists, otherwise ask in the prompt, and validate locally in every case. Allow a bounded repair attempt that shows the model its validation error, then fail with a typed error.
• Record which mode was used, whether the first answer validated, and how many repairs it took. Note schema-subset differences between providers in the shape sheet.
• Done when: schema-invalid and truncated outputs are never returned as success, repairs show up in cost and latency, and the harness can report first-try validity per route.

M5. Embeddings. Uses: Module 08; background in Track 08, Module 02 and Module 05.

• A normalised embeddings call: batching, input-size limits, retries, and each vector returned with provider, model and dimension attached. Use embeddings for something real inside the gateway (for example similarity of a request to labelled examples as a routing feature in M8).
• Guard the vector space: a fallback must never return a vector from a different model than the caller's index expects. Either disable embedding failover or make it explicit.
• Done when: a test proves that fallback cannot mix models, and embedding cost is accounted for as input-only.

M6. Rate limits, retries, timeouts and fallback. Uses: Module 16, Module 17.

• Classify every failure as retryable (transient 429, overload, 5xx, timeout, connection reset) or not (auth, validation, refusal, and quota, billing or spend-cap errors that retrying cannot fix). Honour a retry hint as a minimum wait, add jitter, otherwise back off exponentially, and bound both attempts and total time.
• Exactly one retry layer. OpenAI's rate-limit guide warns that nested retry loops multiply requests; if an SDK retries on its own, switch that off or count it.
• A local limiter that uses the documented rate-limit headers, a per-route circuit breaker, and a fallback chain that is filtered by capability (never fall back to a route that lacks tools, schema support or enough context) and by policy (a sensitive request never falls back to a remote route).
• Done when: a fault matrix covering every injected fault from the stand-in section produces the behaviour you documented, a spend-cap-style 429 without a retry hint is not retried, and a concurrent burst does not turn into a retry storm.

M7. Cost accounting and spend cap. Uses: Module 16, with Track 01, Module 12.

• Cost per attempt is input tokens × input rate + output tokens × output rate, plus any other billing classes the provider's pricing page lists (check them; do not assume). Use provider-reported usage when present and a labelled estimate when not (a cut stream, a local route). Rates come from dated configuration.
• A ledger with one row per attempt, including attempts that failed but may still bill, and a documented decision for the on-device route (free, a shadow price from energy and hardware, or something else, stated).
• Enforce the cap with a reservation: before sending, reserve a worst case (estimated input plus the maximum allowed output at the output rate); after the response, settle to the actual cost. Refuse with a typed error rather than exceed the cap. A missing price means refuse or flag, never count as zero.
• Done when: many concurrent requests against a tiny cap never overshoot by more than a tolerance you state, the cap survives a restart, and three requests recomputed by hand match the ledger.

M8. Routing policies, including the on-device route. Uses: Modules 04, 05, 12, 14, 15, 17.

• A router as a pluggable policy over request features (required capabilities, estimated size, task tag, sensitivity, deadline), the route catalogue, live statistics and remaining budget. Implement at least four policies, for example: fixed to one provider (one baseline per provider), cheapest capable route, latency-aware (observed percentiles with decay), cascade (cheap first, escalate on validation failure or an explicit uncertainty signal), and local-first for sensitive or offline requests.
• The on-device route has its own constraints: smaller context, different speed, no network, possibly weaker tool or schema support. Measure its latency yourself, check that the model fits your memory with the Track 05 arithmetic, and record the quantization you chose and why.
• Done when: policies swap by configuration, every response carries the chosen route and the reason, routing is deterministic for the same inputs and statistics snapshot, and a sensitive request fails with a typed error rather than going remote when the local route is down.

M9. Evaluation harness. Uses: everything above; Modules 04 and 16 for what differs between routes and what it costs.

• The labelled task set. At least 40 items in at least five categories: closed-answer questions, extraction to a schema, tasks that need a tool, classification or triage, long-context lookup, and sensitive-tagged items (include a few unanswerable and injection cases). Each item records an id, category, input, grader and expected result, required capabilities, sensitivity, and a difficulty label assigned before any policy runs. Split it into development and test sets; tune on development, report on test.
• Graders. Programmatic first: normalised exact match, schema validity plus field equality, tool-call trace correctness. If you use a model as a judge anywhere, say so and say what could bias it.
• Runner. The same tasks through each policy, seeded, with record-and-replay (responses stored by request hash) so a run can be repeated offline.
• Metrics. Success rate, first-try schema validity, cost per task and per successful task, latency (median, p95, time to first token), retries and fallbacks per request, cap refusals, local-route share. Use paired comparison on identical items and report uncertainty (for example an interval from resampling).
• Planted-difference check. Prove the harness can detect a difference: a deliberately worse policy, or a stand-in scripted to fail more often on hard items, must come out measurably worse.
• Done when: replay reproduces the results table exactly, you can say which differences are within noise, and the write-up states what stand-in results can and cannot show.

M10. Observability, security pass and report. Uses: all modules; prompting craft belongs to Track 07.

• One structured log line per request and per attempt: correlation id, route, routing reason, status class, latency, tokens, cost, the provider's request id (OpenAI documents x-request-id; Anthropic documents a request-id header). No raw prompts or secrets by default; sampled debug capture is opt-in and redacted. A summary command: spend by route, error rate, fallback rate, p95.
• Security pass: keys only from the environment or a secret store, never in configuration files, logs, error messages, fixtures or history; loopback bind by default plus caller authentication; request size limits; a per-route table of which fields leave the machine.
• Report (at most four pages): decision first (your default policy and why), architecture and boundaries, the shape-sheet register, the results table with uncertainty, the fault-matrix results, the cap test, what you did not test live, what would change your recommendation, and the first thing you would measure before real traffic.

Constraints

• Python. The standard library is enough for stand-ins and for the core; you may use SDKs, an HTTP library and a schema validator, but the retry, routing, cost and fallback logic must be yours.
• No real key anywhere in the repository, tests, logs, transcripts or screenshots. Read keys from the environment; commit a template with placeholder names only. If one leaks, revoke it.
• Every number carries one tag: sourced (URL and date), measured (with the setup), estimated (with the formula), or assumed (with the reason). A number with no tag is a defect.
• Provider facts come from pages you fetched and dated. Prices and model identifiers are configuration inputs with a source and a date, or clearly labelled placeholders. Do not call any provider or model "the best", "the fastest" or "the cheapest"; say what you compared, on which task set, and when.
• If you make live calls: a throwaway key, a provider-side spend limit if the provider offers one, and your own cap set to a trivial amount. Record the date and model identifiers with every live result.
• Results from stand-ins are labelled as stand-in results everywhere they appear.

Non-goals

• A user interface, multi-tenant accounts, billing or user management.
• A benchmark or ranking of providers or models. Your task set is small; it can compare policies, not crown a vendor.
• An agent framework. The tool loop is small and bounded on purpose.
• A vector database or retrieval pipeline (that is Track 08); embeddings here are a gateway call and a router input.
• Training, fine-tuning or prompt optimisation.
• Production deployment, autoscaling or high availability.
• Emulating features a provider lacks. Expose them where they exist and refuse them where they do not.

How to know you've done it well

A report with at least one surprising or disappointing result honestly reported (a policy that lost to the baseline, a fallback that was never safe to use, a cap that overshot until you fixed it) is stronger than one where everything worked.

Suggested extensions

• Response caching keyed by a request hash, and a check of whether your providers price cached input differently.
• A learned router (nearest labelled examples by embedding, or a small classifier trained on your own evaluation logs) compared against your hand-written policies; watch for leakage between development and test items.
• A batch or offline route for evaluation runs, if a provider documents one, and its effect on cost per task.
• Hedged requests (send to a second route after a delay) and a measurement of what the latency gain costs.
• A reasoning-model route with its extra token classes in the ledger (Module 05).
• Vision or audio inputs as a capability that the router must respect (Module 09).
• A model-catalogue check that lists the models each provider currently exposes and flags configured identifiers that have disappeared.
• Shadow traffic: send a copy of requests to a candidate policy and compare, without returning its answers.
• Per-caller budgets and quotas with their own reports.
• A small outbound-fetch tool and an egress allow-list, then attack it with your own planted instructions.

Self-review checklist

Interview questions

Further reading & sources

Provider documentation (all fetched and checked Oct 2026; the official pages are the source of truth and may have changed since):

• API Overview | OpenAI API Reference - OpenAI's reference overview: bearer authentication, the x-request-id and x-ratelimit-* response headers, and where to start for the endpoints behind the OpenAI adapter and stand-in.
• Rate limits | OpenAI API - the rate-limit headers, Retry-After behaviour, backoff guidance, spend limits and the warning about nested retries that milestones M6 and M7 build on.
• Function calling | OpenAI API - the documented tool-call round trip (call id, tool result message) that the M3 loop and its OpenAI-shaped stand-in must reproduce.
• Messages - Claude API Reference - Anthropic's Messages endpoint reference: the request and response fields, stop reasons and content blocks behind the second adapter.
• Streaming messages - Claude Platform Docs - the server-sent event names, partial tool-input JSON and in-stream error events for the M2 parser and stand-in.
• Claude API errors - Claude Platform Docs - the status-code and error-envelope taxonomy, request ids, and the spend-cap 429 without a retry hint that the M6 classifier must handle.
• Gemini API reference | Google AI for Developers - the Gemini API overview: x-goog-api-key authentication, generateContent and streamGenerateContent, and the other interfaces to choose between.
• llama.cpp HTTP server README - the open-weight server's documented OpenAI-compatible endpoints and options, one way to run the on-device route in M8.

Next

Track 07: Prompt Engineering