Module 17: Capstone Project — Train, Benchmark, and Report on a Domain Tokenizer

What this capstone is

Modules 00-16 built four tokenization algorithms from scratch and by library, measured real vocabulary-size and multilingual tradeoffs, found genuine security-relevant special-token behaviors, and quantified cost and context-window consequences. This capstone asks you to put all of it together on one real, self-chosen text domain: train your own tokenizer, benchmark it honestly against production tokenizers, and write up what you found — including the results that don't flatter your own tokenizer.

There is no toy corpus provided. The value of this exercise depends entirely on using real text with real quirks.

The project

1. Pick a real domain corpus

Something with genuine internal structure and enough volume to train on meaningfully (a few hundred KB to a few MB of plain text is enough): your own codebase's source files, a technical field you work in (medical, legal, a specific programming ecosystem), a language other than English, or a mix of code and prose (e.g., this very repository's genai/ markdown files). Do not use a generic English corpus — the whole point is testing whether a domain-specific tokenizer earns its keep, which only shows up against domain-specific text.

2. Train a tokenizer, and justify the algorithm choice

Using sentencepiece (module 05, module 06) or the tokenizers library, train a BPE and a Unigram LM tokenizer on your corpus at a vocabulary size you choose and justify (module 08's tradeoffs). Report:

  • Why you picked that vocabulary size (tie it to module 08's embedding-cost tradeoff, not just "it seemed reasonable").
  • The measured tokens-per-word (or tokens-per-line, for code) for both algorithms on a held-out sample of your corpus not used in training.
  • Which algorithm won on this specific corpus, and your best explanation for why, grounded in modules 03-06's mechanics — not a guess.

3. Benchmark against real production tokenizers

Using tiktoken's cl100k_base and at least one Hugging Face tokenizer (module 07's pattern), measure tokens-per-word on the same held-out sample for every tokenizer, your own included. Build a table like module 07's, and answer honestly: did your domain-specific tokenizer actually beat the general-purpose ones on your own corpus? By how much? If it didn't win, say so — a negative result reported honestly is more valuable than a flattering one, and is common: general-purpose tokenizers trained on enormous, diverse corpora are a genuinely high bar to beat with a small custom training run.

4. Stress-test for the pitfalls from module 13

Run your own tokenizer against:

  • A handful of numbers, and check whether digit-chunking is place-value-aligned or exhibits the same context-dependent inconsistency verified in module 13.
  • A handful of rare/unusual strings from your corpus (proper nouns, identifiers, or symbols that appear only once or twice) — check whether any produce disproportionately many tokens (a sign of under-coverage) or, for WordPiece specifically, an outright [UNK] (module 05's whole-word failure mode).
  • If your corpus includes any non-Latin-script text, measure and report its chars-per-token ratio (module 14) alongside the Latin-script portion's, and compare the gap to what cl100k_base showed in module 14's cross-language experiment.

5. Cost and context-window impact, quantified

Using module 12's cost-calculator pattern and module 11's context-window arithmetic, compute: for a realistic request against your domain (a prompt built from a chunk of your corpus, sent to a real API), how much does the tokenizer choice change the actual token count, and therefore the actual cost and the fraction of a real context window consumed? Express this as a concrete number ("switching from cl100k_base to my trained tokenizer changes a typical 2,000-word document from N tokens to M tokens, a K% difference") — not a general impression.

6. Write-up

A short document (1-3 pages) covering, in order: the corpus and why you chose it, the vocabulary-size decision and its justification, the benchmark table (your tokenizer vs. at least two production ones), the pitfall stress-test findings, the cost/context-window impact, and a final honest verdict: was training a custom tokenizer worth it for this specific corpus, or would you recommend using an off-the-shelf one in practice? Both answers are legitimate outcomes.

Constraints

  • Every quantitative claim must come from code you actually ran on your actual corpus — no illustrative or assumed numbers. This capstone is specifically about replacing every earlier module's example numbers with your own, freshly measured ones.
  • Use a held-out sample (text your tokenizer wasn't trained on) for every benchmark comparison in sections 2-3 — training-set performance overstates real-world tokenization quality and would make the comparison meaningless.
  • Keep the write-up under 3 pages. Compressing your findings into a clear verdict is part of the exercise.

How to know you've done it well

  • Someone who hasn't read this track could follow your benchmark table and reach the same conclusion you did.
  • You can point to a specific module (03-16) that explains why each of your measured results came out the way it did — not just what the number was.
  • Your write-up includes at least one result that surprised you, or didn't go the way you expected before running the experiment.
  • The final verdict is a real recommendation, not a hedge — "it depends" is only acceptable if you specify exactly what it depends on, backed by the numbers you measured.

Next

Track 02: Transformer Architecture