# Module 04: Queries, Keys and Values
## Why this matters
Track 00 module 05 already introduced queries, keys, and values as
roles — "what am I looking for," "what do I match against," "what do I
actually retrieve." This module verifies the specific, concrete reason
those three roles are computed as **three separate learned linear
projections** of the same input, rather than using the raw token
vectors directly for all three. The reason isn't stylistic: using a
token's own embedding as both its query and its key creates a
mathematically provable, structural bias — verified directly below —
that separate projections exist specifically to break.
## Concepts
### Q, K, V as three projections of the same input
Every token's vector `x` produces three different vectors via three
separate learned weight matrices (module 01's matrix multiplication,
applied here):
```
Q = X @ Wq (queries: "what am I looking for")
K = X @ Wk (keys: "what do I offer, for matching")
V = X @ Wv (values: "what do I offer, for retrieval")
```
If `Wq`, `Wk`, and `Wv` were all the identity matrix (or, equivalently,
if there were no projections at all — `Q = K = V = X`), attention would
be scoring each token's raw embedding against every other token's raw
embedding directly. This is the case worth testing explicitly, because
it has a real, provable problem.
### Verified: without separate projections, every token is biased toward attending to itself
For any vector `x`, its dot product with itself is `x . x = |x|^2`.
By the Cauchy-Schwarz inequality, `x . y <= |x| * |y|` for *any* other
vector `y`, with equality only when `y` is exactly parallel to `x`.
This means: if `Q = K = X` (no separate projections), **every token's
raw self-similarity score is mathematically guaranteed to be at least
as large as its similarity to any other token** — an unavoidable bias
toward self-attention, baked in purely by the geometry of dot products,
verified directly:
```python
import numpy as np
np.random.seed(1)
d = 8
X = np.random.randn(5, d)
scores_noproj = X @ X.T # Q = K = X directly, no projection
print(np.round(scores_noproj, 2))
print(all(np.argmax(scores_noproj[i]) == i for i in range(5)))
```
Verified output — the diagonal (each token's score with itself) is
**always the largest value in its row**:
```
[[14.11 5.53 -2.79 0.59 0.59]
[ 5.53 9.29 -0.59 1.74 0.37]
[-2.79 -0.59 4.73 -0.03 4.19]
[ 0.59 1.74 -0.03 3.16 -0.97]
[ 0.59 0.37 4.19 -0.97 6.25]]
True # every row's argmax is on the diagonal
```
Every one of the 5 tokens scores highest against *itself* — not because
that's meaningful for the task, but because it's a mathematical
guarantee of using the same vector as both query and key.
### Verified: separate Q/K projections break this bias
Now the same 5 tokens, but with two independently random weight
matrices standing in for two separately *learned* projections:
```python
Wq = np.random.randn(d, d)
Wk = np.random.randn(d, d)
Q, K = X @ Wq, X @ Wk
scores_proj = Q @ K.T
print(np.round(scores_proj, 2))
print(all(np.argmax(scores_proj[i]) == i for i in range(5)))
```
Verified output:
```
[[ 10.1 -9.59 -23.47 -21.91 -17.86]
[-11.87 1.25 7.72 -22.57 -2.89]
[ -8.71 15.24 7.72 10.15 5.63]
[-10.47 -7.67 -14.7 0.13 -17.8 ]
[ -9.2 3.6 10.32 0.95 7.41]]
False # NOT every row's argmax is on the diagonal anymore
```
With separate projections, only 1 of the 5 tokens still scores highest
against itself (row 0) — the mathematical guarantee is gone. `Wq` and
`Wk` project the same input into two *different* vector spaces, so
`Q_i . K_i` (a token scored against its own key) is no longer
constrained by Cauchy-Schwarz to be the maximum — the model is free to
learn that a given token should mostly attend to *other* tokens, which
is exactly what's needed for attention to do anything useful (a
sentence where every word mostly just looks at itself wouldn't be
learning any real cross-token relationships at all).
### Why three separate roles, not just two
Query and key together decide *how much* attention flows where; value
decides *what content* actually gets retrieved once that's decided.
Splitting these into three separate learned spaces (rather than, say,
reusing `K` as `V`) lets the model learn a genuinely different
specialization for "what makes me a good match for this query" versus
"what information should be handed over once I've been matched" — a
token's role as a *matching target* and its role as *retrieved content*
don't have to encode the same information. This is a real, used
distinction — the model may find it useful for a value vector to carry
rich content-specific information while its corresponding key vector
carries something much simpler, purely optimized for being found by the
right queries.
## Reference
```
Vector Computed as Role Used in
─────── ───────────── ─────────────────────────── ────────────────
Query X @ Wq "what am I looking for" scores = Q @ K.T
Key X @ Wk "what do I offer for scores = Q @ K.T
matching"
Value X @ Wv "what do I offer for output = weights @ V
retrieval"
```
```
Configuration Self-attention score guarantee (verified)
──────────────────── ────────────────────────────────────────────
Q = K = X (no projection) Every token's self-score is >= its score
against any other token (Cauchy-Schwarz) —
verified: 5/5 tokens scored highest on
themselves
Q = X@Wq, K = X@Wk No such guarantee — verified: only 1/5
(separate, learned) tokens still scored highest on themselves
```
## Hands-on exercises
### Exercise 1 — reproduce the self-attention-bias proof
Run both the `Q=K=X` and `Q=X@Wq, K=X@Wk` versions above with your own
random seed and vector count. Confirm the no-projection version always
puts every row's maximum on the diagonal, and the projected version
usually doesn't.
### Exercise 2 — verify the Cauchy-Schwarz guarantee directly
For a single vector `x`, compute `x . x` and compare it against `x . y`
for several random `y` vectors of the same dimension. Confirm `x . x`
is always at least as large, and find (or construct) the one case where
they're equal (hint: what does `y` have to be relative to `x`?).
### Exercise 3 — test whether the bias reappears if `Wq == Wk`
Repeat the projected-scores experiment, but deliberately set `Wq` and
`Wk` to be the *same* matrix (still a real projection, applied to both
Q and K identically). Confirm the diagonal-maximum bias reappears —
connecting the fix specifically to `Wq` and `Wk` being *different*
learned matrices, not merely to the presence of *some* projection.
## Independent challenge
A researcher proposes a simplified attention variant that ties
`Wq = Wk` to save parameters (only one projection matrix to learn
instead of two), arguing "the model can still learn useful attention
patterns, just with a constraint." Using this module's verified
Cauchy-Schwarz finding, write two or three sentences on what specific
behavior this constraint would bias the model toward, and whether that
seems like a reasonable simplification for most tasks.
Discussion
Tying `Wq = Wk` reintroduces exactly the self-attention bias verified
above (exercise 3's setup) — for any token, `Q_i . K_i = (x_i @ W) . (x_i
@ W)`, which by the same Cauchy-Schwarz argument is guaranteed to be at
least as large as `Q_i . K_j` for any other token `j`, since both Q and
K come from applying the *same* transformation to the same input.
Whether that's reasonable depends heavily on the task: for tasks where
strong self-reference is actually useful (a token largely re-confirming
its own identity, similar to a copy-mechanism), the constraint could be
harmless or even helpful; for the wide range of tasks where meaningful
cross-token relationships (a pronoun genuinely needing information from
elsewhere) are the whole point, forcing every token to default toward
attending to itself is a real cost, not just a parameter-saving
convenience.
Answers
1. `x . x = |x|^2` is guaranteed to be at least as large as `x . y` for
any other vector `y` of the same norm or less specific relationship
— precisely, `x . y <= |x| * |y|`, with equality only when `y` is
exactly parallel to `x`.
2. Every one of the 5 tokens scored highest against itself — the
diagonal was the row-maximum in all 5 rows, exactly as the
Cauchy-Schwarz guarantee predicts when `Q = K = X`.
3. The guarantee disappeared — verified: only 1 of 5 tokens still
scored highest against itself once `Q = X@Wq` and `K = X@Wk` used
two independently different projection matrices.
4. Reintroduces it — verified in exercise 3: since `Q_i = x_i@W` and
`K_i = x_i@W` both come from the *same* transformation applied to
the same input, the Cauchy-Schwarz argument still applies and the
diagonal-maximum bias returns.
5. Because keys decide *how much* attention a token receives (a
matching role) while values decide *what content* is actually
retrieved once matched (a content role) — these are different jobs
the model may want to specialize differently, so giving them
independent learned projections lets the model optimize each role
separately rather than being forced to conflate them.