Module 02: Why Not Characters, Why Not Words

Why this matters

Module 01 established that text is really a stream of bytes. Module 03 will introduce BPE, the algorithm real tokenizers use to group those bytes into tokens. Between the two sits an obvious question this module answers properly: why group them at all? Why not just feed the model individual characters, or go the other direction and use whole words?

This isn't a throat-clearing exercise — both of those simpler options were tried, extensively, and both lose to sub-word tokenization for specific, quantifiable reasons. Knowing those reasons is what makes BPE's design in module 03 feel inevitable rather than arbitrary, and it's also directly useful: you'll recognize "why not just split on spaces" as a question with a real, measured answer the next time someone proposes it.

Concepts

Option 1: character-level tokenization

Every character (or, more precisely after module 01, every byte) is its own token. Vocabulary size: maybe 256 (one per byte value).

 "unbelievable" character-level:

 u  n  b  e  l  i  e  v  a  b  l  e
 │  │  │  │  │  │  │  │  │  │  │  │
 12 SEPARATE TOKENS for one word

 PRO: tiny vocabulary (~256), never an unknown token
 CON: sequences become very long

Why this loses: track 02 will show that a transformer's attention cost grows quadratically with sequence length (O(n²)). Character-level tokenization inflates sequence length by roughly 4-5x versus word-level for English — turning a 500-word document into a ~2,500-token sequence instead of ~650. That's not a minor inefficiency; it's a massive, avoidable multiplier on the single most expensive part of the whole architecture.

There's a second cost, more subtle: the model has to do more work per unit of meaning. Predicting "u", then "n", then "b"... to spell "unbelievable" spends many forward passes rediscovering a pattern (this word's spelling) that a coarser unit could have captured in one step.

Option 2: word-level tokenization

Split on whitespace and punctuation. One token per word.

 "unbelievable" word-level:  [unbelievable]     -- 1 token

 PRO: short sequences, each token carries real meaning
 CON: the vocabulary problem, below

This sounds strictly better until you meet the out-of-vocabulary (OOV) problem. A word-level vocabulary is necessarily finite — you fix it at training time, typically tens of thousands of entries. Any word not in that set has no representation at all.

 vocabulary built from training data:
   the, cat, sat, run, running, walk, ...

 NEW WORD AT INFERENCE TIME:  "COVID-19"
   (didn't exist when a pre-2019 vocabulary was built)

     ┌───────────────────────────────────────┐
     │ WORD-LEVEL: no slot for it. Replaced   │
     │ with a single generic <UNK> token --   │
     │ ALL information about the word is LOST │
     └───────────────────────────────────────┘

 also unrepresentable this way: typos, made-up brand
 names, rare technical jargon, most non-English words,
 and anything coined after the vocabulary was frozen

The <UNK> (unknown) token is the crux of the failure. Once a word maps to <UNK>, the model has literally no signal left about what that word was — "COVID-19", "asdfghjkl", and "Kubernetes" (if none were in the training vocabulary) become indistinguishable. This is not a rare edge case; language constantly mints new words, and a word-level model goes stale the day a term it never saw becomes common.

The word-level vocabulary size problem, independent of OOV

Even ignoring <UNK>, word-level vocabularies are large and inefficient in a specific way: English morphology means related words are separate, unrelated-looking vocabulary slots.

 run, runs, running, ran, runner, runners, reruns, ...

 word-level: 7+ SEPARATE, unrelated vocabulary entries,
             each needing its own learned embedding,
             with NO shared representation between them

 (a sub-word tokenizer, module 03, would likely represent
  most of these as "run" + a small, REUSED suffix piece —
  one shared root, several cheap variations)

A word-level vocabulary large enough to cover a language's real productivity (compounding, inflection, technical terminology) balloons into the hundreds of thousands of entries — each one an extra row in the model's embedding table and output layer, at real memory and compute cost, for morphological variants that could have shared structure.

The gap that sub-word tokenization fills

                    sequence length        vocabulary /
                    (cost: quadratic         OOV handling
                     attention, O(n^2))
                    ─────────────────      ─────────────────
 character-level     ✗ very long            ✓ perfect (~256
                        (12 tokens for          tokens, never
                        "unbelievable")         unknown)

 word-level           ✓ short                 ✗ broken on any
                        (1 token)                unseen word

 SUB-WORD (BPE)       between the two         between the two —
 (module 03)          — reuses common          common words are
                       long tokens for          1 token, RARE/NEW
                       frequent words           words gracefully
                                               split into known
                                               pieces, never <UNK>

The actual result, previewed from module 00: "Tokenization" split into Token + ization — a real, frequent word (Token) stayed as one token, and a productive suffix (ization, which also appears in "organization", "realization", ...) became a reusable second token instead of forcing the whole compound word into a brand-new vocabulary slot or destroying it as <UNK>.

Crucially, this also solves the OOV problem outright: because the vocabulary includes individual bytes as a fallback (module 01), a sub-word tokenizer can always fall back to spelling out any truly novel string byte by byte. There is no <UNK> token in a modern GPT-family tokenizer, because there's no input it's structurally unable to represent — only inputs it represents less efficiently than common English text.

Reference

Hands-on exercises

Standard library only.

1. Measure the sequence-length gap yourself

text = "Tokenization strategies fundamentally determine how efficiently a transformer processes language."

char_tokens = list(text.replace(" ", ""))   # rough character-level, ignoring spaces
word_tokens = text.split()

print(f"characters: {len(char_tokens)} tokens")
print(f"words:      {len(word_tokens)} tokens")
print(f"ratio:      {len(char_tokens)/len(word_tokens):.1f}x longer at character level")

Confirm the ratio is in the 4-6x range this module claims. Then connect it to the O(n²) attention cost from track 02 module 05: if sequence length goes up 5x, by what factor does attention's compute cost grow?

2. Build a tiny word-level tokenizer and break it

vocab = {"the": 0, "cat": 1, "sat": 2, "on": 3, "mat": 4, "<UNK>": 5}

def tokenize(text, vocab):
    return [vocab.get(w, vocab["<UNK>"]) for w in text.lower().split()]

print(tokenize("the cat sat on the mat", vocab))          # all known
print(tokenize("the cat sat on the rug", vocab))           # "rug" is new
print(tokenize("the dog ran across the yard", vocab))      # mostly new

Look at the second and third outputs. Multiple completely different unknown words all produce the identical <UNK> ID. Write one sentence on what information is destroyed by this collision, and why a model downstream cannot recover it.

3. Quantify morphological waste

family = ["run", "runs", "running", "ran", "runner", "runners", "rerun", "reruns"]
print(f"{len(family)} distinct words, ALL sharing the root 'run'")
print(f"word-level: {len(family)} separate vocabulary slots, "
      f"{len(family)} separate embeddings to learn")

# a plausible sub-word split (illustrative, not real BPE output yet)
subword_guess = {
    "run": ["run"], "runs": ["run", "s"], "running": ["run", "ning"],
    "ran": ["ran"], "runner": ["run", "ner"], "runners": ["run", "ner", "s"],
    "rerun": ["re", "run"], "reruns": ["re", "run", "s"],
}
pieces = set()
for toks in subword_guess.values():
    pieces.update(toks)
print(f"sub-word: only {len(pieces)} distinct pieces needed: {sorted(pieces)}")

State in one sentence why fewer distinct pieces, reused across many words, is a genuine efficiency win — not just a smaller number, but what it means for how much the model has to learn about each piece.

4. Find real out-of-vocabulary casualties

Pick five words that plausibly didn't exist, or weren't common, before 2020 — brand names, slang, technical terms, or invented product names. Using the toy tokenize() function from exercise 2 (extend vocab with ~30 common English words of your choice, but deliberately omit your five test words), confirm all five collapse to the same <UNK> ID.

Then look up how a real BPE tokenizer (tiktoken, from module 00) handles the same five words, and report what you find. This is the concrete payoff of "no <UNK> token" — write one sentence on the difference.

5. Diagnose and fix: a translation system stuck in 2018

A team maintains a word-level machine translation system. Every few months, translation quality visibly degrades on customer text, and the fix has always been "retrain with an updated vocabulary." They ask why this keeps happening and whether it's avoidable.

Independent challenge

Take a real corpus (a few pages of any text — code, prose, or a mix) and build both a character-level and a naive word-level tokenizer over it (using Python's str.split() is fine for the word-level one). For each:

1. Report total token count and vocabulary size.
2. Simulate the OOV problem: hold out 20% of the text as a "test set" not seen when building the word-level vocabulary, then measure what fraction of test-set words are OOV.
3. Compute the sequence-length ratio between the two approaches on the same text.

Write a short paragraph stating, in your own words and using your own numbers, why sub-word tokenization exists as a compromise between them — you should be able to derive module 03's motivation entirely from your own measurements.

Common mistakes & troubleshooting

• Assuming "split on whitespace" is a reasonable tokenizer. It breaks on any word not seen during vocabulary construction, silently and totally (exercise 2).
• Underestimating the sequence-length cost of character-level tokenization. It's not a minor inefficiency — combined with O(n²) attention, a 5x longer sequence can mean a ~25x higher attention cost.
• Believing a bigger word-level vocabulary solves OOV. It only delays it — language keeps producing new words faster than any fixed list can be updated (exercise 5).
• Not recognizing morphological waste. Treating "run", "runs" and "running" as three unrelated vocabulary slots wastes both vocabulary space and the model's ability to share what it learns about "run" across all its forms.

Checkpoint quiz

1. Name the two costs of character-level tokenization.
2. What is the out-of-vocabulary problem, and why is <UNK> a destructive fix rather than a real one?
3. Why does a word-level vocabulary grow so large for a morphologically rich language, even ignoring OOV entirely?
4. Why can a sub-word tokenizer never produce an <UNK> token?
5. In one sentence, state where sub-word tokenization sits between the character-level and word-level extremes.

Further reading & sources

• Neural Machine Translation of Rare Words with Subword Units (Sennrich et al., 2016) - the paper that brought byte-pair encoding into NLP specifically to solve the OOV problem this module describes; read before module 03.
• Hugging Face: Summary of the tokenizers - a clear side-by-side of word-level, character-level and sub-word approaches with the same tradeoffs framed slightly differently.
• Google's Neural Machine Translation System (Wu et al., 2016) - section 4 documents Google Translate's own move away from word-level vocabularies for exactly the OOV reasons in this module, at production scale.
• Character-Level Language Modeling with Deeper Self-Attention (Al-Rfou et al., 2018) - a serious attempt at pure character-level transformer language modeling; useful for seeing the sequence-length cost taken to its logical extreme.
• tiktoken (GitHub) - confirms hands-on in exercise 4 that a real production tokenizer never emits an <UNK>-equivalent.

Next

Module 03: Byte-Pair Encoding Explained