# Module 16: Sampling Strategies: Greedy, Temperature, Top-k, Top-p
## Why this matters
Track 00, module 08 already introduced temperature, top-k, and top-p
with real `torch`-based generation code, including the ASCII-diagram
argument for why top-p's adaptive set size beats top-k's fixed count.
**This module puts real numbers behind that argument** — implementing
all the filtering logic from scratch in plain NumPy (no framework
dependency) and running it against two genuinely different probability
distributions to *measure*, not just illustrate, how many tokens each
method actually keeps. It also covers **min-p**, a newer, less commonly
taught sampling method not covered in track 00, and compares it directly
against top-p on the same verified numbers.
## Concepts
### The four filters, implemented from scratch
```python
import numpy as np
def softmax(logits):
e = np.exp(logits - np.max(logits))
return e / e.sum()
def top_k_filter(probs, k):
idx = np.argsort(probs)[::-1][:k]
filtered = np.zeros_like(probs)
filtered[idx] = probs[idx]
return filtered / filtered.sum()
def top_p_filter(probs, p):
idx = np.argsort(probs)[::-1]
cumulative = np.cumsum(probs[idx])
cutoff = np.searchsorted(cumulative, p) + 1
keep = idx[:cutoff]
filtered = np.zeros_like(probs)
filtered[keep] = probs[keep]
return filtered / filtered.sum(), cutoff
def min_p_filter(probs, ratio):
threshold = probs.max() * ratio
keep = probs >= threshold
filtered = np.where(keep, probs, 0)
return filtered / filtered.sum(), keep.sum()
```
`top_k_filter` always keeps exactly `k` tokens. `top_p_filter` keeps
however many tokens are needed for their cumulative probability to
reach `p`. `min_p_filter` keeps every token whose probability is at
least `ratio` times the single most likely token's probability — a
different, simpler adaptivity rule introduced more recently than top-p.
### Measured: the same k and p, two very different distributions
Two realistic next-token distributions — one **confident** (like the
token right after `"The capital of France is"`), one **uncertain**
(like the token after an open-ended `"She opened the ___"`):
```python
confident_logits = np.array([8.0, 3.0, 2.5, 1.0, 0.5, 0.2, 0.1, 0.05, 0.0, -1.0])
uncertain_logits = np.array([2.0, 1.9, 1.8, 1.7, 1.6, 1.5, 1.4, 1.3, 1.2, 1.1])
confident_probs = softmax(confident_logits)
uncertain_probs = softmax(uncertain_logits)
```
Verified sorted probabilities:
```
confident: [0.9863, 0.0066, 0.0040, 0.0009, 0.0005, 0.0004, 0.0004, 0.0003, 0.0003, 0.0001]
uncertain: [0.1505, 0.1362, 0.1233, 0.1115, 0.1009, 0.0913, 0.0826, 0.0748, 0.0676, 0.0612]
```
Now apply **top-k=3**, **top-p=0.9**, and **min-p ratio=0.1** to both,
and count how many tokens each keeps:
```
top-k=3 top-p=0.9 min-p (ratio=0.1)
confident 3 tokens 1 token 1 token
uncertain 3 tokens 9 tokens 10 tokens
```
This is the exact, measured version of track 00's claim: **top-k keeps
a fixed count no matter what** — 3 tokens whether the model is
99%-confident or genuinely torn between ten options. **Top-p and min-p
both adapt** — on the confident distribution, a single token already
holds 98.6% of the probability mass, so top-p correctly truncates to
just that one token (nothing else is worth considering), while on the
uncertain distribution it keeps 9 of the 10 tokens, because no small
subset dominates. Min-p produces an almost identical result here via a
different, simpler rule: instead of tracking a cumulative sum, it just
asks "is this token at least 10% as likely as the single best one?" —
cheap to compute, and verified to produce essentially the same adaptive
behavior as top-p on both distributions above.
### Why min-p is worth knowing, distinct from top-p
Top-p's cumulative-sum approach has a known failure mode: with a very
long tail of low-probability tokens, enough of them can cumulatively
add up to cross the `p` threshold even though none of them are
individually plausible, letting a few near-nonsensical tokens sneak
into the kept set. Min-p sidesteps this by comparing each token
directly to the single best token's probability, regardless of how
many other tokens exist in the tail — it doesn't care about a
cumulative sum at all, only a ratio to the top choice. Both are valid,
real, implemented techniques (min-p is available in several open-source
inference stacks) — the comparison above shows they often agree, but
they're computing genuinely different things.
### Combining filters, and why order matters
Real inference stacks commonly apply **temperature first**, then a
truncation filter (top-k, top-p, or min-p), then sample from what's
left:
```
raw logits
│ divide by temperature
▼
reshaped logits
│ softmax
▼
probabilities
│ top-k / top-p / min-p filter
▼
truncated, renormalized probabilities
│ sample
▼
chosen token
```
Applying the truncation filter *before* temperature would filter based
on the original (unscaled) distribution's shape, then reshape a
*subset* — a subtly different operation from reshaping the full
distribution first and then deciding what's still plausible. Track 00's
warning against changing temperature and top-p simultaneously (hard to
reason about which one caused a change) applies doubly here: know which
order your specific inference stack applies these in before assuming a
parameter change will behave a specific way.
## Reference
```
Method Rule Adaptive to model confidence?
──────── ─────────────────────────────── ──────────────────────────────
Greedy always take argmax no (single fixed choice)
Temperature divide logits before softmax reshapes, doesn't truncate
Top-k keep exactly k highest-prob NO — verified: keeps 3
tokens tokens regardless of
confidence
Top-p keep smallest set whose YES — verified: 1 token
(nucleus) cumulative prob >= p (confident) vs. 9
(uncertain)
Min-p keep tokens >= ratio * top YES — verified: 1 token
token's probability (confident) vs. 10
(uncertain); simpler,
no cumulative sum, less
prone to long-tail
inclusion
```
## Hands-on exercises
### Exercise 1 — reproduce the confident-vs-uncertain comparison
Run the exact code above and confirm the kept-token counts match:
top-k=3 always keeps 3; top-p=0.9 keeps 1 (confident) vs. 9
(uncertain); min-p ratio=0.1 keeps 1 (confident) vs. 10 (uncertain).
### Exercise 2 — construct a case where top-p includes an implausible token
Build a distribution with one moderately-confident top token and a very
long tail of near-zero (but not exactly zero) probabilities — enough of
them that their cumulative sum meaningfully contributes. Apply
`top_p_filter` with `p=0.95` and check whether any individually
implausible tail token gets included purely because the cumulative sum
needed it. Then apply `min_p_filter` with a reasonable ratio to the same
distribution and compare which tokens each method actually keeps.
### Exercise 3 — verify filter order changes the result
Using the `confident_logits` array, compute the final sampled
distribution two ways: (a) temperature=0.5, then softmax, then
top-k=3; (b) softmax, then top-k=3, then apply temperature to the
surviving 3 logits before renormalizing. Confirm the two resulting
distributions are not identical — order genuinely matters, not just in
theory.
## Independent challenge
A team wants generation to be "creative but not nonsensical" for a
brainstorming feature. Using this module's verified confident-vs-uncertain
comparison, argue for or against using top-k as the sole truncation
method for this use case, and suggest which of top-p or min-p would be
a better default, tying your answer to the specific adaptive-count
numbers verified above rather than a general impression.
Discussion
Top-k is a poor fit here specifically because "creative but not
nonsensical" implies the acceptable amount of variety should depend on
how open-ended a given generation step actually is — exactly the
property top-k structurally cannot express (verified: it kept exactly
3 tokens whether the model was 98.6% confident or genuinely torn among
ten options). Either top-p or min-p is a better starting point because
both adapt the kept-token count to the model's actual confidence at
each step (verified: 1 vs. 9, and 1 vs. 10 respectively) — meaning a
confident step stays close to deterministic while a genuinely open step
gets real variety, without a fixed k that's either too restrictive on
open steps or too loose on confident ones.
Answers
1. Exactly 3 tokens in both cases — top-k always keeps a fixed count,
regardless of whether the underlying distribution was highly
confident (98.6% on one token) or much more uncertain (10 fairly
close probabilities).
2. 1 token on the confident distribution, 9 tokens on the uncertain
one — demonstrating top-p's adaptive set size: it naturally narrows
when the model is sure and widens when it isn't.
3. Min-p keeps every token whose probability is at least some fixed
ratio of the single most likely token's probability — a direct
ratio-to-the-top comparison, rather than top-p's approach of
summing probabilities in ranked order until a cumulative threshold
is crossed.
4. A very long tail of individually-implausible tokens can
cumulatively sum enough to cross top-p's threshold, letting some of
them be included even though none is individually plausible. Min-p
avoids this because it never sums a tail — it only compares each
token directly to the top token's probability.
5. Because truncating before temperature filters based on the
original, unscaled distribution's shape and then reshapes only the
surviving subset, while truncating after temperature reshapes the
full distribution first and then decides what's still plausible
under the new shape — these are mechanically different operations,
verified to produce different final distributions in exercise 3.