# Module 18: Capstone Project
## Roadmap
**What this capstone is.** Track 06 gave you the provider landscape one slice at a time: a first OpenAI call
([Module 00](../00-the-openai-api-your-first-call/README.md)), Chat Completions
([Module 01](../01-chat-completions-deep-dive/README.md)), the Responses API and statefulness
([Module 02](../02-the-responses-api-and-statefulness/README.md)), streaming
([Module 03](../03-streaming-responses/README.md)), model families and generations
([Module 04](../04-model-families-and-generations/README.md)), reasoning models
([Module 05](../05-reasoning-models-and-test-time-compute/README.md)), function calling
([Module 06](../06-function-calling-basics/README.md)), structured outputs and JSON Schema
([Module 07](../07-structured-outputs-and-json-schema/README.md)), the embeddings API
([Module 08](../08-the-embeddings-api/README.md)), vision and audio inputs
([Module 09](../09-vision-and-audio-inputs/README.md)), the Anthropic Claude API
([Module 10](../10-the-anthropic-claude-api/README.md)), the Google Gemini API
([Module 11](../11-the-google-gemini-api/README.md)), open-weight models
([Module 12](../12-open-weight-models-llama-mistral-qwen/README.md)), the Hugging Face ecosystem
([Module 13](../13-the-hugging-face-ecosystem/README.md)), small language models
([Module 14](../14-small-language-models/README.md)), on-device and edge inference
([Module 15](../15-on-device-and-edge-inference/README.md)), pricing, rate limits and quotas
([Module 16](../16-pricing-rate-limits-and-quotas/README.md)), and provider abstraction and model routing
([Module 17](../17-provider-abstraction-and-model-routing/README.md)). This capstone asks you to put all of it
behind one door: a **multi-provider assistant gateway** that your own applications call instead of calling
providers directly.
It also leans on earlier tracks: token cost economics
([Track 01, Module 12](../../01-tokens-and-language-modeling/12-token-cost-economics/README.md)) and context
windows ([Track 01, Module 11](../../01-tokens-and-language-modeling/11-context-windows-explained/README.md)) for
the cost model, embedding models and embedding pipelines
([Track 08, Module 02](../../08-embeddings-and-vector-databases/02-embedding-models-compared/README.md),
[Module 05](../../08-embeddings-and-vector-databases/05-generating-embeddings-at-scale/README.md)), quantization
and GGUF ([Track 04, Module 15](../../04-training-fine-tuning-and-alignment/15-quantization-int8-int4-gguf/README.md)),
and the memory and CPU arithmetic for the on-device route
([Track 05, Module 05](../../05-gpus-and-genai-hardware/05-calculating-a-models-memory-requirements/README.md),
[Module 14](../../05-gpus-and-genai-hardware/14-running-models-on-cpu-and-apple-silicon/README.md)).
There is no dataset, no solution, no code given and no single correct design. Two good gateways can draw their
boundaries differently, choose different default policies and still both be right. What makes yours good is that
every provider fact is sourced and dated, every failure mode has a test, every cost is accounted for, and the claim
"my routing policy is better" rests on evidence you produced.
**Afterwards you can** build, test and defend the layer that sits between application code and model providers:
normalising several APIs, surviving rate limits and outages, capping spend, routing by capability, cost and
latency, and measuring whether a routing change helped. That layer exists in most real LLM systems, and it is where
most production incidents (a surprise bill, a retry storm, an injected tool result) actually happen.
**An honesty rule for this capstone.** You may have no API keys. If you build local stand-ins, your engineering
(parsing, streaming, tool loop, retries, caps, routing, evaluation) is genuinely exercised, but the model behind a
stand-in is scripted, so no number from it says anything about any real model or provider. Label every result that
came from a stand-in as such.
## Concept
> [!key]
> A gateway is only worth having if you can show, with a task set you labelled yourself, that its routing, fallback
> and spend controls beat "call one model" on cost and latency without losing correctness, and that it fails safely
> when a provider, a tool or the budget misbehaves.
## Visual diagram
A decomposition of the problem, not a prescribed design; your boundaries may differ.
```
caller (CLI or local HTTP) sends ONE normalised request:
messages, optional tools, optional schema, task tags, sensitivity flag, deadline, budget
│
▼
┌─ GATEWAY (your code) ───────────────────────────────────────────────────────────────────┐
│ │
│ ┌───────────┐ ┌────────────┐ ┌──────────────┐ ┌───────────────────────────────┐ │
│ │ ROUTER │──►│ SPEND GATE │──►│ RESILIENCE │──►│ ADAPTERS │ │
│ │ policy on │ │ reserve the│ │ timeout, │ │ normalise request, stream, │ │
│ │ capability│ │ worst case;│ │ retry, rate │ │ tool calls, structured output,│ │
│ │ cost, │ │ refuse at │ │ limiter, │ │ embeddings, errors, usage │ │
│ │ latency, │ │ the cap │ │ fallback │ │ │ │
│ │ sensitive │ │ │ │ │ │ │ │
│ └─────▲─────┘ └─────┬──────┘ └──────┬───────┘ └───────────────┬───────────────┘ │
│ │ │ settle │ every attempt │ usage, status │
│ │ ▼ ▼ │ │
│ ┌─────┴─────────────────────────────────────────────────────────┐ │ │
│ │ LEDGER + LOGS + METRICS: route, reason, status, tokens, cost, │◄──┘ │
│ │ latency, provider request id; feeds router stats and caps │ │
│ └───────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────────────┘
│ only adapters touch the network
▼
┌────────────────┐ ┌────────────────┐ ┌────────────────────────┐
│ provider A │ │ provider B │ │ on-device / open-weight│
│ real or │ │ real or │ │ route (local server, │
│ local stand-in │ │ local stand-in │ │ in-process, or stand-in│
└────────────────┘ └────────────────┘ └────────────────────────┘
Milestones, in build order (each ends with an offline test):
M0 contracts ─► M1 adapters ─► M2 streaming ─► M3 tool loop ─► M4 schema ─► M5 embeddings ─┐
┌──────────────────────────────────────────────────────────────────────────────────────────┘
└─► M6 retries, limits, fallback ─► M7 cost + cap ─► M8 routing ─► M9 evaluation ─► M10 report
M9 is how you learn whether a change (a policy, a retry rule, a model id) helped; if it cannot
tell two policies apart, the task set is too small. M10 also holds the security pass and the logs.
```
## Project
### The problem
A small product team ships four features that all need a language model: a support-ticket triager that must return
JSON in a fixed shape, a chat helper that must stream, an order-assistant that calls internal tools (order lookup,
unit conversion), and a help-centre search that needs embeddings. Today each feature calls one provider's SDK
directly. Then a price changes, a provider has a bad afternoon, one feature gets rate-limited at peak, an
experiment runs up an unexpected bill, and the privacy lead says tickets marked sensitive must never leave the
company's machines.
You build the one internal service (a library plus a CLI, or a small local HTTP server; your choice) that all four
features call instead.
### Who uses it
| User | What they need from the gateway |
|---|---|
| Feature developer | One request and response shape; typed, predictable errors; opt-in streaming, tools, schema and embeddings; can see which route answered, but does not have to care |
| Budget owner | A spend cap that holds under concurrency; cost per request and per route; retries that never silently double the bill |
| Privacy-minded caller | Requests tagged sensitive never leave the machine; the route taken is auditable afterwards |
| Whoever tunes routing (you, next month) | A harness that says whether a new policy is better, with evidence and uncertainty |
### Providers, keys and stand-ins
Requirements: at least **two providers whose documented wire formats differ** (two endpoints of the same API
family count as one), plus **at least one on-device or open-weight route**. Real keys are optional.
If you have no keys, or want tests that never touch a network, build a **local stand-in per provider dialect**: a
small HTTP server bound to `127.0.0.1` that implements the subset of that provider's *documented* shapes you use.
A workable method:
1. Fetch the provider's official reference and guide pages yourself (starting points are under Further reading)
and note the date you read them.
2. For each endpoint you use, write a **shape sheet** row: method and path, auth header, required request fields,
the response fields you read (text, tool calls, usage, finish reason), streaming framing and event names, error
envelope, rate-limit headers, request-id header. OpenAI also publishes a machine-readable specification in the
[`openai/openai-openapi`](https://github.com/openai/openai-openapi) repository, which is a good cross-check for
the OpenAI-shaped rows.
3. Make the scripted "model" deterministic and say so in the stand-in's name and in every log line it produces.
4. Add **fault injection** you can trigger per request: 429 with and without a retry hint, an overload or 5xx
response, a slow response, a stream cut mid-way, an error event after a 200, malformed tool-call arguments, an
unknown tool name, schema-invalid output, truncated output, and a 401.
5. Check the stand-in against the docs by re-reading them, not against your memory. If you can make even one live
call to a real endpoint, diff the real response shape against your stand-in; the differences are findings.
What the official pages said when this module was written (checked Oct 2026; the docs are the source of truth and
may have changed since, so re-read before you rely on any row):
| Provider | Auth header | Streaming | Error and limit signals you can reproduce in a stand-in |
|---|---|---|---|
| OpenAI | `Authorization: Bearer` | `stream` on; Chat Completions sends `delta` chunks, the Responses API sends typed events such as `response.output_text.delta` and `response.completed` | `x-request-id`; `x-ratelimit-*` headers; `Retry-After` may be present on temporary 429 and 503; quota and billing errors should not be retried |
| Anthropic | `x-api-key` plus `anthropic-version` | Server-sent events named `message_start`, `content_block_start`, `content_block_delta`, `content_block_stop`, `message_delta`, `message_stop`, plus `ping` and in-stream `error` events; tool inputs arrive as partial JSON strings | JSON error envelope with a `request_id`; statuses including 429, 500, 504 and 529; a spend-cap 429 has no `retry-after` and keeps failing until access resumes |
| Gemini | `x-goog-api-key` | `streamGenerateContent` over server-sent events | Limits are per project and measured in requests per minute, tokens per minute and requests per day; spend-based limits return `429 RESOURCE_EXHAUSTED` ([rate limits page](https://ai.google.dev/gemini-api/docs/rate-limits)) |
| Local open-weight server | Optional, for example an `--api-key` option | llama.cpp's server documents OpenAI-compatible `/v1/chat/completions` and `/v1/embeddings` endpoints | Local failure modes (model still loading, out of memory, slow decode) are yours to observe and reproduce |
Providers can document more than one way to do the same job (OpenAI has Chat Completions and the Responses API;
the Gemini reference lists both an Interactions API and `generateContent`). Pick one interface per provider,
record why, and do not assume a new one behaves like the old one.
### Milestones
Each milestone ends with a test that runs against stand-ins only: offline, deterministic, no keys. "Uses" names the
Track 06 modules to reread; the exact depth you need is your call.
**M0. Contracts, configuration and the shape sheet.**
Uses: Modules [00](../00-the-openai-api-your-first-call/README.md), [01](../01-chat-completions-deep-dive/README.md),
[04](../04-model-families-and-generations/README.md), [16](../16-pricing-rate-limits-and-quotas/README.md).
- Write the shape sheet (dated URLs). Define, in prose first, your normalised request, response, usage, finish
reasons and error classes (retryable or not; auth, validation, quota, overload, timeout, refusal).
- Put routes, model identifiers, capability flags (tools, schema mode, streaming, embeddings, context and output
limits) and prices in configuration, each with a source URL and a date. Nothing perishable in code.
- Done when: changing a model id, a price or a capability flag needs no code change, and a test fails loudly when a
route is missing a field the router depends on.
**M1. Normalised adapters, non-streaming.**
Uses: Modules [01](../01-chat-completions-deep-dive/README.md), [02](../02-the-responses-api-and-statefulness/README.md),
[10](../10-the-anthropic-claude-api/README.md), [11](../11-the-google-gemini-api/README.md),
[12](../12-open-weight-models-llama-mistral-qwen/README.md), [17](../17-provider-abstraction-and-model-routing/README.md).
- One interface, one adapter per dialect plus one for the on-device route. Map system instructions, multi-turn
history, output-length limits, stop reasons and usage. Keep the provider's request id and raw finish reason next
to the normalised ones.
- Decide, and write down, what you refuse to normalise (features only one provider has) and how a caller reaches
them: an escape hatch, or a typed capability error.
- If you use official SDKs, hide them behind your interface and account for their built-in retries (see M6).
- Done when: one caller test suite passes against every adapter, unknown extra response fields never crash parsing,
and a "leaky abstraction log" lists each difference you hit and your decision.
**M2. Streaming.**
Uses: [Module 03](../03-streaming-responses/README.md), plus the streaming parts of Modules
[10](../10-the-anthropic-claude-api/README.md) and [11](../11-the-google-gemini-api/README.md).
- A normalised event stream (text delta, tool-call fragment, usage, finish, error). Parse each provider's
server-sent-event framing yourself, ignore event types you do not know, and reassemble tool-call arguments that
arrive in fragments.
- Measure time to first token and total time. Propagate client disconnects upstream.
- Decide what a failure after output has started means for retry, fallback (never splice two models' text without
saying so) and cost (partial usage still costs something).
- Done when: a stand-in stream cut mid-way yields a typed error with the partial text preserved and cost recorded,
and an error event after a 200 is handled like any other failure.
**M3. Tool-calling loop.**
Uses: [Module 06](../06-function-calling-basics/README.md), with each provider's tool shapes from Modules
[01](../01-chat-completions-deep-dive/README.md), [10](../10-the-anthropic-claude-api/README.md) and
[11](../11-the-google-gemini-api/README.md).
- A tool registry (name, description, parameter schema, handler, timeout, read-only or side-effecting). A loop that
calls the model, runs the requested tools (in parallel when several are requested), returns each result tied to
the right call id, and repeats until a final answer or a limit.
- Validate arguments before running anything. Unknown tool, bad JSON, handler exception and timeout become results
the model can see, or a hard stop, by an explicit policy.
- Treat tool output as untrusted data: size-limit it, keep it out of the instruction channel, allow-list tools per
request, and gate side-effecting tools. Do not rely on the model to resist an injected instruction.
- Done when: the loop cannot exceed its step, time or spend budget, it works with two providers' tool shapes, and a
stand-in scripted to *obey* an instruction planted in tool output still cannot cause a disallowed tool to run.
**M4. Schema-validated structured output.**
Uses: [Module 07](../07-structured-outputs-and-json-schema/README.md); prompting side in
[Track 07: Prompt Engineering](../../07-prompt-engineering/README.md).
- The caller supplies a schema. Use a provider-enforced mode where the route's capability flags say one exists,
otherwise ask in the prompt, and **validate locally in every case**. Allow a bounded repair attempt that shows the
model its validation error, then fail with a typed error.
- Record which mode was used, whether the first answer validated, and how many repairs it took. Note schema-subset
differences between providers in the shape sheet.
- Done when: schema-invalid and truncated outputs are never returned as success, repairs show up in cost and
latency, and the harness can report first-try validity per route.
**M5. Embeddings.**
Uses: [Module 08](../08-the-embeddings-api/README.md); background in
[Track 08, Module 02](../../08-embeddings-and-vector-databases/02-embedding-models-compared/README.md) and
[Module 05](../../08-embeddings-and-vector-databases/05-generating-embeddings-at-scale/README.md).
- A normalised embeddings call: batching, input-size limits, retries, and each vector returned with provider, model
and dimension attached. Use embeddings for something real inside the gateway (for example similarity of a request
to labelled examples as a routing feature in M8).
- Guard the vector space: a fallback must never return a vector from a different model than the caller's index
expects. Either disable embedding failover or make it explicit.
- Done when: a test proves that fallback cannot mix models, and embedding cost is accounted for as input-only.
**M6. Rate limits, retries, timeouts and fallback.**
Uses: [Module 16](../16-pricing-rate-limits-and-quotas/README.md),
[Module 17](../17-provider-abstraction-and-model-routing/README.md).
- Classify every failure as retryable (transient 429, overload, 5xx, timeout, connection reset) or not (auth,
validation, refusal, and quota, billing or spend-cap errors that retrying cannot fix). Honour a retry hint as a
minimum wait, add jitter, otherwise back off exponentially, and bound both attempts and total time.
- Exactly one retry layer. OpenAI's rate-limit guide warns that nested retry loops multiply requests; if an SDK
retries on its own, switch that off or count it.
- A local limiter that uses the documented rate-limit headers, a per-route circuit breaker, and a fallback chain
that is filtered by capability (never fall back to a route that lacks tools, schema support or enough context) and
by policy (a sensitive request never falls back to a remote route).
- Done when: a fault matrix covering every injected fault from the stand-in section produces the behaviour you
documented, a spend-cap-style 429 without a retry hint is not retried, and a concurrent burst does not turn into a
retry storm.
**M7. Cost accounting and spend cap.**
Uses: [Module 16](../16-pricing-rate-limits-and-quotas/README.md), with
[Track 01, Module 12](../../01-tokens-and-language-modeling/12-token-cost-economics/README.md).
- Cost per attempt is `input tokens × input rate + output tokens × output rate`, plus any other billing classes the
provider's pricing page lists (check them; do not assume). Use provider-reported usage when present and a labelled
estimate when not (a cut stream, a local route). Rates come from dated configuration.
- A ledger with one row per attempt, including attempts that failed but may still bill, and a documented decision
for the on-device route (free, a shadow price from energy and hardware, or something else, stated).
- Enforce the cap with a **reservation**: before sending, reserve a worst case (estimated input plus the maximum
allowed output at the output rate); after the response, settle to the actual cost. Refuse with a typed error
rather than exceed the cap. A missing price means refuse or flag, never count as zero.
- Done when: many concurrent requests against a tiny cap never overshoot by more than a tolerance you state, the cap
survives a restart, and three requests recomputed by hand match the ledger.
**M8. Routing policies, including the on-device route.**
Uses: Modules [04](../04-model-families-and-generations/README.md), [05](../05-reasoning-models-and-test-time-compute/README.md),
[12](../12-open-weight-models-llama-mistral-qwen/README.md), [14](../14-small-language-models/README.md),
[15](../15-on-device-and-edge-inference/README.md), [17](../17-provider-abstraction-and-model-routing/README.md).
- A router as a pluggable policy over request features (required capabilities, estimated size, task tag,
sensitivity, deadline), the route catalogue, live statistics and remaining budget. Implement at least four
policies, for example: fixed to one provider (one baseline per provider), cheapest capable route, latency-aware
(observed percentiles with decay), cascade (cheap first, escalate on validation failure or an explicit
uncertainty signal), and local-first for sensitive or offline requests.
- The on-device route has its own constraints: smaller context, different speed, no network, possibly weaker tool
or schema support. Measure its latency yourself, check that the model fits your memory with the Track 05
arithmetic, and record the quantization you chose and why.
- Done when: policies swap by configuration, every response carries the chosen route and the reason, routing is
deterministic for the same inputs and statistics snapshot, and a sensitive request fails with a typed error
rather than going remote when the local route is down.
**M9. Evaluation harness.**
Uses: everything above; Modules [04](../04-model-families-and-generations/README.md) and
[16](../16-pricing-rate-limits-and-quotas/README.md) for what differs between routes and what it costs.
- **The labelled task set.** At least 40 items in at least five categories: closed-answer questions, extraction to
a schema, tasks that need a tool, classification or triage, long-context lookup, and sensitive-tagged items
(include a few unanswerable and injection cases). Each item records an id, category, input, grader and expected
result, required capabilities, sensitivity, and a difficulty label **assigned before any policy runs**. Split it
into development and test sets; tune on development, report on test.
- **Graders.** Programmatic first: normalised exact match, schema validity plus field equality, tool-call trace
correctness. If you use a model as a judge anywhere, say so and say what could bias it.
- **Runner.** The same tasks through each policy, seeded, with record-and-replay (responses stored by request hash)
so a run can be repeated offline.
- **Metrics.** Success rate, first-try schema validity, cost per task and per *successful* task, latency (median,
p95, time to first token), retries and fallbacks per request, cap refusals, local-route share. Use paired
comparison on identical items and report uncertainty (for example an interval from resampling).
- **Planted-difference check.** Prove the harness can detect a difference: a deliberately worse policy, or a
stand-in scripted to fail more often on hard items, must come out measurably worse.
- Done when: replay reproduces the results table exactly, you can say which differences are within noise, and the
write-up states what stand-in results can and cannot show.
**M10. Observability, security pass and report.**
Uses: all modules; prompting craft belongs to [Track 07](../../07-prompt-engineering/README.md).
- One structured log line per request and per attempt: correlation id, route, routing reason, status class,
latency, tokens, cost, the provider's request id (OpenAI documents `x-request-id`; Anthropic documents a
`request-id` header). No raw prompts or secrets by default; sampled debug capture is opt-in and redacted. A
summary command: spend by route, error rate, fallback rate, p95.
- Security pass: keys only from the environment or a secret store, never in configuration files, logs, error
messages, fixtures or history; loopback bind by default plus caller authentication; request size limits; a
per-route table of which fields leave the machine.
- Report (at most four pages): decision first (your default policy and why), architecture and boundaries, the
shape-sheet register, the results table with uncertainty, the fault-matrix results, the cap test, what you did not
test live, what would change your recommendation, and the first thing you would measure before real traffic.
### Constraints
- Python. The standard library is enough for stand-ins and for the core; you may use SDKs, an HTTP library and a
schema validator, but the retry, routing, cost and fallback logic must be yours.
- No real key anywhere in the repository, tests, logs, transcripts or screenshots. Read keys from the environment;
commit a template with placeholder names only. If one leaks, revoke it.
- Every number carries one tag: sourced (URL and date), measured (with the setup), estimated (with the formula), or
assumed (with the reason). A number with no tag is a defect.
- Provider facts come from pages you fetched and dated. Prices and model identifiers are configuration inputs with a
source and a date, or clearly labelled placeholders. Do not call any provider or model "the best", "the fastest"
or "the cheapest"; say what you compared, on which task set, and when.
- If you make live calls: a throwaway key, a provider-side spend limit if the provider offers one, and your own cap
set to a trivial amount. Record the date and model identifiers with every live result.
- Results from stand-ins are labelled as stand-in results everywhere they appear.
### Non-goals
- A user interface, multi-tenant accounts, billing or user management.
- A benchmark or ranking of providers or models. Your task set is small; it can compare *policies*, not crown a
vendor.
- An agent framework. The tool loop is small and bounded on purpose.
- A vector database or retrieval pipeline (that is Track 08); embeddings here are a gateway call and a router input.
- Training, fine-tuning or prompt optimisation.
- Production deployment, autoscaling or high availability.
- Emulating features a provider lacks. Expose them where they exist and refuse them where they do not.
### How to know you've done it well
| Area | Needs work | Meets | Strong |
|---|---|---|---|
| Correctness | Happy path on one provider; outputs differ silently across providers; invalid structured output returned as success | One caller suite passes on every adapter; tool results matched to call ids; every structured result validated locally; streaming and non-streaming agree on text and usage | Every provider difference has a recorded decision (normalise, expose or refuse); stream assembly and schema validation have property-style tests |
| Robustness | Fixed-sleep retries; every error retried; fallback ignores capabilities | Errors classified; retry hints honoured; bounded attempts and deadline; one retry layer; capability- and policy-filtered fallback; fault matrix passes | Circuit breaker, defined mid-stream failure policy, retry-storm test under concurrency, behaviour traced to each provider's documented error taxonomy |
| Cost control | Cost computed afterwards from a hard-coded price; cap checked after the call | Dated price configuration; per-attempt ledger including retries; worst-case reservation before sending; cap holds under a concurrency test | Cap survives restarts; per-caller budgets; mid-stream abort at the cap; local-route cost policy stated; provider-side limit used as a backstop; report reproducible from the ledger alone |
| Routing and evaluation | A few hand-picked prompts; one run; no uncertainty | 40 or more labelled items, development/test split, four or more policies, programmatic graders, paired comparison with intervals, replay reproduces the table | Planted-difference check passes; cost and latency shown against success (a trade-off view, not one score); results broken out by category; honest limits on what stand-ins prove |
| Observability | Print statements; logs cannot be tied to attempts | Structured log per request and attempt with correlation id, route, reason, status, tokens, cost, latency and provider request id; a summary command | "Why did request X go to route Y and what did it cost?" answerable from logs alone; redacted sampled capture; spend and error-rate thresholds |
| Testability | Tests need live keys or are flaky | Whole suite runs offline against stand-ins, deterministically, with fault injection, from one command | Stand-ins checked against the shape sheet; optional live contract tests behind a flag with the cap on; a deliberately broken retry or cap makes a test fail |
| Security | Keys in code, config or logs; tool output pasted into prompts as trusted; no tool allow-list | Keys from environment only and never echoed; loopback bind plus caller auth; size limits; tool output treated as untrusted and size-limited; per-request tool allow-list; side-effecting tools gated; sensitive requests never routed remote | Injection tests where the scripted model obeys the planted text and the gateway still holds; canary-secret test over logs and errors; least-privilege keys; per-route data-flow table |
| Communication | Reader cannot tell what was checked against a source | Decision first, four pages or fewer, tagged numbers, shape-sheet register, results with uncertainty | States what would change the recommendation, what was not tested live, and what to measure first |
A report with at least one surprising or disappointing result honestly reported (a policy that lost to the baseline,
a fallback that was never safe to use, a cap that overshot until you fixed it) is stronger than one where
everything worked.
### Suggested extensions
- Response caching keyed by a request hash, and a check of whether your providers price cached input differently.
- A learned router (nearest labelled examples by embedding, or a small classifier trained on your own evaluation
logs) compared against your hand-written policies; watch for leakage between development and test items.
- A batch or offline route for evaluation runs, if a provider documents one, and its effect on cost per task.
- Hedged requests (send to a second route after a delay) and a measurement of what the latency gain costs.
- A reasoning-model route with its extra token classes in the ledger ([Module 05](../05-reasoning-models-and-test-time-compute/README.md)).
- Vision or audio inputs as a capability that the router must respect ([Module 09](../09-vision-and-audio-inputs/README.md)).
- A model-catalogue check that lists the models each provider currently exposes and flags configured identifiers
that have disappeared.
- Shadow traffic: send a copy of requests to a candidate policy and compare, without returning its answers.
- Per-caller budgets and quotas with their own reports.
- A small outbound-fetch tool and an egress allow-list, then attack it with your own planted instructions.
### Self-review checklist
> [!check]
> - I can run the whole test suite with no network and no keys, and it passes the same way twice.
> - Every provider fact in my shape sheet has a URL and the date I read it; none came from memory.
> - Changing a model id, a price or a capability flag needs no code change.
> - For each fault in my matrix I can point to the test and to the log line showing what the gateway did.
> - No retry layer exists that I have not accounted for (SDK retries are off or counted).
> - A request that needs tools, a schema or a long context never falls back to a route that lacks them.
> - Retries and failed attempts appear in the ledger, and the cost report reproduces from the ledger alone.
> - With a tiny cap and many concurrent requests, total spend stays within the tolerance I stated.
> - Structured output is validated locally on every route, including routes with a provider-enforced mode.
> - The tool loop is bounded by steps, time and spend, and tool output is treated as data.
> - Embedding fallback cannot mix vector spaces.
> - Sensitive requests never reach a remote route in any test, including when the local route is down.
> - A search of the repository, logs and fixtures for key-shaped strings and my canary secrets finds nothing.
> - The task set was labelled before any policy ran, development and test are separate, and replay reproduces my table.
> - My comparison shows uncertainty, and a planted-difference check shows the harness can tell a worse policy apart.
> - The report labels stand-in results as such and says what I did not test live.
> - At least one result surprised or disappointed me, and I reported it.
## Interview questions
> [!interview]
> Your gateway gets a 429 from a provider. Walk me through exactly what it does, and tell me which 429s must not be
> retried.
What a strong answer covers
- Classify before retrying: a transient rate limit is retryable; a quota, billing or spend-cap condition is not,
because waiting does not fix it. OpenAI's guide says quota and billing errors should not be retried and that
`Retry-After` may appear on temporary 429 and 503 responses. Anthropic's docs describe a tier spend-cap 429 with no
`retry-after` that keeps failing until access resumes.
- Use the retry hint as a minimum wait, add jitter, otherwise exponential backoff, and bound both attempts and total
deadline. One retry layer only, since an SDK retrying inside your retry multiplies requests.
- Decide between waiting and falling back to another route, filtering the fallback by capability and policy
(sensitivity, tools, schema, context size).
- Account for cost and side effects: failed attempts may still bill, and a repeated tool call may repeat a side
effect. Log every attempt with the provider's request id.
- Surface a typed error to the caller when you give up, instead of a generic failure.
What a strong answer covers
- Evidence from a labelled task set written before the comparison, with a development/test split so the policy is
not tuned on what it is judged on.
- Compare on identical items (paired), on cost per *successful* task rather than per call, and on latency tails, not
only means. Report uncertainty and say which differences are within noise.
- Break results out by category: a mean can hide a regression on exactly the hard or sensitive items that matter.
- Count the hidden costs of the cheap path: escalations, repairs, retries and fallbacks.
- Be honest about the evidence's limits (small set, one date, specific model versions), pin identifiers, re-run when
a model or price changes, and consider shadow traffic or a canary before a full switch.
- Stand-in results validate the harness, not the policy's real-world quality.
What a strong answer covers
- Assume the model might comply; the defence cannot rely on it refusing. OWASP's
[LLM01 Prompt Injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) entry describes indirect
injection through external content and recommends least-privilege access and human approval for high-risk
actions.
- Controls outside the model: a per-request tool allow-list, no tool that can read secrets or send data out unless
explicitly granted, gates on side-effecting and outbound actions, tool output treated as size-limited data rather
than instructions, and no keys or environment contents anywhere the model or a tool can reach.
- Verification: a test where the scripted stand-in obeys the planted text and the gateway still refuses to run the
disallowed action, plus logs showing the blocked attempt.
- Related hygiene: keys from the environment only, redaction, a canary-secret test over logs and error messages.