Module 18: Capstone Project — Build, Break, and Report on a Mini Transformer
What this capstone is
Modules 00-17 built and verified every piece of a transformer in isolation — path length, dot products, Q/K/V, masking, positional encoding, the FFN, residual connections, layer normalization — then assembled them into a model that actually trains (module 17). This capstone asks you to do something none of the previous modules did: deliberately break individual architectural components, one at a time, and measure what actually happens — turning this track's many individually-verified claims ("residual connections prevent gradient collapse," "positional encoding is necessary for order-sensitivity") into a single, coherent set of ablation experiments run on one model you build and train yourself.
There is no toy corpus provided, no solution given, and no fixed list of ablations you must run beyond the required set below — the point is running real experiments and reporting real, possibly surprising results.
The project
1. Pick a real, small text corpus
Something with genuine structure worth learning: your own writing, a
public-domain book (a few chapters), a technical document, or — a good
option if nothing else is at hand — this repository's own genai/
markdown content. A few hundred KB is enough. Character-level or a
simple word-level tokenizer (track 01) both work; state which you used
and why.
2. Build and train a working baseline model
Using module 17's pattern (module 15's MiniTransformerLM plus a
training loop), train a baseline model on your corpus until loss drops
substantially and clearly below the ln(vocab_size) sanity-check value
(module 15). Report: your architecture's d_model, num_heads,
num_layers, d_ff; your final training loss; and a sample of
generated text, honestly reported (including any repetition or
incoherence you observe, not just a cherry-picked good example).
3. Run the required ablations, and measure real numbers for each
For each ablation, train an otherwise-identical model (same hyperparameters, same corpus, same number of training steps, same random seed where possible) with exactly one change, and report the resulting training loss curve and a generated sample:
- No positional encoding. Remove the positional embedding entirely (module 09-11). Predict, before running it, what you expect to happen, then run it and compare against your prediction.
- No residual connections. Remove the
x +in each sub-layer (module 13). Predict the expected effect on training (module 13's verified gradient-collapse finding at depth) before running it. - No causal mask (bidirectional self-attention on a next-token prediction task). Remove the mask (module 07) while keeping the next-token-prediction training objective. Predict what you expect — this is a genuinely interesting case, since the "cheating" module 07 warned about becomes directly measurable here.
- A meaningfully shallower or deeper model (module 15's depth experiment, but on your own real corpus instead of a toy one).
4. Write up what actually happened, including surprises
For each ablation, report: what you predicted before running it (using this track's verified findings from the relevant module), what actually happened, and — where they differ — your best explanation for the gap. A prediction that turned out wrong is a more valuable finding than one that matched perfectly; report both kinds honestly.
5. One deliberate architectural change of your own choosing
Beyond the required ablations, make one change this track didn't
explicitly test — a different d_ff/d_model ratio (module 12), a
different positional scheme (RoPE instead of sinusoidal, or vice versa,
modules 10-11), tied vs. untied embeddings (module 02), or something
else you're curious about. Measure and report its effect the same way.
Constraints
- Every reported number must come from a real run on your actual corpus — no illustrative or assumed results. This capstone is specifically about generating your own evidence, not reusing this track's example numbers.
- Keep total compute reasonable — a CPU-trainable model (module 17's scale, or modestly larger) run for a few hundred to a few thousand steps per configuration is enough to see real, measurable differences; there's no need for GPU-scale training to get honest results.
- State your prediction for each required ablation before running it, in your write-up, so the comparison between prediction and outcome is genuine rather than reconstructed after the fact.
How to know you've done it well
- Someone who's read this track but not run your experiments could follow your report and understand both what you expected and what actually happened, for every ablation.
- At least one ablation's actual result surprised you relative to your stated prediction — and you can offer a real explanation for the gap, grounded in a specific module's mechanics, not a guess.
- Your report distinguishes between "this ablation made training
fail outright" (loss stays near
ln(vocab_size)or diverges) and "this ablation made training measurably worse but still functional" (loss decreases, just less than the baseline) — these are different outcomes worth reporting precisely, not collapsed into "it got worse." - The no-causal-mask ablation specifically addresses whether the model is visibly "cheating" (module 07) — reaching an unusually low loss very quickly, precisely because it can see the answer it's supposed to predict.