# Module 00: What Is a Token
## Why this matters
Every weird, specific behavior you've noticed in a chat model traces back
to one fact: **the model never sees words — it sees tokens**, and a token
is not "a word." It's a chunk of text (sometimes a whole word, sometimes
a word-piece, sometimes a single character or byte) drawn from a fixed
vocabulary the model learned during training.
This single fact explains things that otherwise look like bugs:
- **Why GPT-family models are famously bad at "count the letters in
'strawberry'."** The model never sees the letters `s-t-r-a-w-b-e-r-r-y`
— it sees a small number of opaque token IDs. Asking it to count letters
is like asking someone to count the pixels in a word they only ever
saw as a single icon.
- **Why API pricing is "per token," not "per word" or "per character,"**
and why the same sentence in Chinese, Hindi, or Arabic can cost
noticeably more tokens than the English equivalent — different
tokenizers were trained on English-majority corpora, so non-Latin
scripts often split into more, smaller tokens.
- **Why "context window" is measured in tokens** (e.g. "128K context"),
and why a document that looks short on screen can silently blow past a
limit once code, whitespace, or an unusual language inflates its token
count.
Track 02 (transformers) will show *why* the architecture operates on
discrete tokens rather than raw characters or continuous audio directly.
For now: get comfortable actually looking at tokens, not just reading
about them.
## Concepts
### A token is a vocabulary entry, not a linguistic unit
A tokenizer is built from a fixed **vocabulary** — a lookup table mapping
text chunks to integer IDs, typically 30,000-100,000+ entries, learned
once from a training corpus (module 03 covers exactly how that vocabulary
gets built). At inference time, tokenizing is just: split the input text
into the longest sequence of vocabulary-recognized chunks, and replace
each with its integer ID.
```
"Tokenization is the first step"
│
▼ split on vocabulary entries
┌────────┬──────────┬─────┬──────┬────────┬───────┐
│ Token │ ization │ is │ the │ first │ step │
└────────┴──────────┴─────┴──────┴────────┴───────┘
│ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼
┌────────┬──────────┬─────┬──────┬────────┬───────┐
│ 12904 │ 2065 │ 374 │ 279 │ 1176 │ 3094 │
└────────┴──────────┴─────┴──────┴────────┴───────┘
the model sees ONLY these integers
NOT one-per-word: "Tokenization" → 2 tokens
NOT one-per-char: "step" → 1 token
note the leading spaces: " is" is ONE token, not " " + "is"
```
```python
import tiktoken
# cl100k_base is the tokenizer used by GPT-3.5/GPT-4-era models;
# o200k_base is used by GPT-4o and newer.
enc = tiktoken.get_encoding("cl100k_base")
text = "Tokenization is the first step of every LLM pipeline."
ids = enc.encode(text)
print(ids)
# [12904, 2065, 374, 279, 1176, 3094, 315, 1475, 445, 11237, 15660, 13]
print(len(ids), "tokens for", len(text), "characters")
# 12 tokens for 55 characters -- NOT one token per word (13 words) and
# NOT one token per character (55 chars). Somewhere in between.
for tid in ids:
print(tid, repr(enc.decode([tid])))
# 12904 'Token'
# 2065 'ization'
# 374 ' is'
# 279 ' the'
# 1176 ' first'
# ...
```
Notice `"Tokenization"` split into `Token` + `ization` — a whole common
word (`the`, `is`, `first`) usually gets exactly one token (with a leading
space folded in — more on that below), while a rarer or compound word
splits into pieces. That splitting behavior is the entire subject of
module 01.
### Tokens, not characters, not words — and the leading-space quirk
Run the exercise above on `" is"` vs `"is"` and you'll get **different
token IDs** — most BPE-based tokenizers (including `cl100k_base`) fold the
preceding space into the token itself, because "a space followed by a
word" is an extremely common pattern worth its own vocabulary entry. This
is a frequent source of confusion when programmatically comparing or
concatenating token sequences — `"Hello" + " world"` does not tokenize the
same as `"Hello world"` tokenized as one string, because how the pieces
get glued back together changes which vocabulary entries match.
### A rough rule of thumb, and why it's only a rough one
For English text, a common approximation is **~4 characters per token**,
or **~0.75 tokens per word**. This is genuinely just a heuristic for quick
mental estimates — it breaks down for:
- **Code**, where punctuation-dense syntax (`{`, `()`, `->`, indentation)
tokenizes very differently from prose.
- **Non-English text**, especially non-Latin scripts, which the dominant
tokenizers (trained on English-heavy corpora) split far less
efficiently — a Hindi or Japanese sentence often costs 2-3x more tokens
than an English sentence saying the same thing. This is a real,
measurable cost and latency disadvantage for non-English-first
products, not a minor curiosity.
- **Numbers**, which some tokenizers split digit-by-digit or in small
groups specifically so arithmetic reasoning has a chance of working
token-by-token.
Always measure, don't estimate, when a real budget (context window, API
cost) is on the line — which is exactly what the exercises below do.
### Special tokens
Beyond text-derived vocabulary entries, every model reserves a handful of
**special tokens** that never appear in ordinary text but carry structural
meaning the model was trained to respect — e.g. an end-of-sequence marker,
a beginning-of-text marker, or (for chat-tuned models) markers delimiting
where a system/user/assistant turn starts and ends. You don't typically
construct these by hand — the chat-completion API layer inserts them for
you — but they count toward your token budget just like any other token,
and library functions like `tiktoken`'s `encode` will refuse to encode
literal special-token-looking text by default (a deliberate safety
guard against a user's message accidentally forging a fake system turn).
## Reference
| Term | Means |
|---|---|
| Token | One vocabulary entry — a chunk of text mapped to an integer ID |
| Vocabulary | The fixed set of all tokens a specific tokenizer recognizes |
| Tokenizer | The algorithm/lookup table that converts text ↔ token IDs |
| `tiktoken` | OpenAI's open-source tokenizer library, usable offline without an API key |
| `cl100k_base` | The tokenizer encoding used by GPT-3.5-turbo/GPT-4-era models |
| `o200k_base` | The tokenizer encoding used by GPT-4o and newer models |
| Special token | A reserved token (e.g. end-of-sequence) with structural rather than linguistic meaning |
| Task | Code |
|---|---|
| Load a tokenizer | `enc = tiktoken.get_encoding("cl100k_base")` |
| Load the tokenizer for a specific model | `enc = tiktoken.encoding_for_model("gpt-4o")` |
| Text → token IDs | `enc.encode(text)` |
| Token IDs → text | `enc.decode(ids)` |
| Count tokens without decoding | `len(enc.encode(text))` |
| Inspect one token's text | `enc.decode([single_id])` |
## Hands-on exercises
Install once: `pip install tiktoken`
### 1. Encode, decode, and look at the pieces
```python
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
text = "The quick brown fox jumps over the lazy dog."
ids = enc.encode(text)
print(f"{len(text)} chars -> {len(ids)} tokens")
for tid in ids:
print(f" {tid:6d} {enc.decode([tid])!r}")
```
Run it. Which words got split into more than one piece, if any? (This
sentence is famously "pangram-simple" English — expect mostly whole-word
tokens. You'll get a very different picture in exercise 3.)
### 2. Prove the leading-space quirk
```python
a = enc.encode("Hello world")
b = enc.encode("Hello") + enc.encode(" world")
c = enc.encode("Hello") + enc.encode("world")
print(a)
print(b)
print(c)
print("a == b:", a == b, " a == c:", a == c)
```
Before running it, predict which pair (if any) will be equal. Then run it
and reconcile any surprise with the "leading-space quirk" concept above.
### 3. English vs. code vs. another language — measure, don't guess
```python
samples = {
"english": "The weather today is sunny with a light breeze.",
"python_code": "def add(a: int, b: int) -> int:\n return a + b",
"hindi": "आज मौसम धूप वाला और हल्की हवा के साथ है।",
}
for label, s in samples.items():
ids = enc.encode(s)
print(f"{label:12s} {len(s):3d} chars {len(ids):3d} tokens "
f"{len(s)/len(ids):.2f} chars/token")
```
Run it (swap in a real sentence in whatever second language you speak, if
not Hindi). Which sample has the worst chars-per-token ratio? Write one
sentence connecting that result to the "non-English cost disadvantage"
point in the concepts section.
### 4. Build a token-budget checker
Write a function `fits_in_budget(text: str, max_tokens: int, encoding: str = "cl100k_base") -> bool`
that returns whether `text` tokenizes to `max_tokens` or fewer. Then use
it to check whether three paragraphs of your choice (paste in any real
text — an email, a README section, a news snippet) fit inside a
1,000-token budget. This is the exact check a RAG pipeline (track 09) runs
before stuffing a retrieved chunk into a prompt.
Reference solution
```python
import tiktoken
def fits_in_budget(text: str, max_tokens: int, encoding: str = "cl100k_base") -> bool:
enc = tiktoken.get_encoding(encoding)
return len(enc.encode(text)) <= max_tokens
```
Answer
Words and tokens are different units — a 50-word string is not
necessarily 50 tokens (per this module, usually more). The fix must
`encode`, slice the ID list to `max_tokens`, then `decode`:
```python
def trim_to_tokens(text: str, max_tokens: int) -> str:
enc = tiktoken.get_encoding("cl100k_base")
ids = enc.encode(text)
return enc.decode(ids[:max_tokens])
```
Answers
1. The model only ever sees token IDs, not individual characters — a word
is usually one or a few opaque tokens, so the letters composing it
aren't directly visible to the model to count.
2. Trick: they usually ARE equal in this specific case, because
`" world"` (with leading space) is itself a valid single vocabulary
token that both forms produce identically — the quirk shows up when
you compare against `encode("Hello") + encode("world")` (no leading
space), which is NOT the same, since `"world"` without a leading space
is a different vocabulary entry.
3. The dominant tokenizers were trained on English-majority corpora, so
non-Latin scripts (or generally underrepresented languages) split into
more, smaller tokens per equivalent meaning.
4. Words and tokens are different units — many common words are one
token, but longer/rarer/compound words split into multiple tokens, so
a word count systematically undercounts real token usage.
5. The API silently adds structural tokens around your raw message
content (turn boundaries, etc.), which count toward context-window and
billing limits even though you never typed them — a raw `encode()`
on your own text alone will undercount the real total.