# Module 13: Residual Connections and Layer Normalization
## Why this matters
Attention (modules 04-11) and the FFN (module 12) are the two
computational sub-layers of a transformer block — but stacking many of
them (module 15 stacks dozens) doesn't work reliably without two
unglamorous but load-bearing additions: a residual (skip) connection
around each sub-layer, and layer normalization. This module verifies
both are doing real, measurable work, not boilerplate: a 30-layer deep
network's gradient at the input **collapses by 8 orders of magnitude**
without residual connections, and layer normalization is confirmed to
force every token's representation to a fixed mean/standard-deviation
regardless of its raw input scale.
## Concepts
### Residual connections: `output = x + sublayer(x)`, not `output = sublayer(x)`
Every attention and FFN sub-layer in a real transformer is wrapped like
this:
```
output = x + Sublayer(x) NOT: output = Sublayer(x)
```
The `+ x` — adding the sub-layer's *input* back to its *output* — is
the residual (or "skip") connection. It looks almost too simple to
matter. Verified directly below, it's the difference between a deep
stack being trainable at all and a deep stack whose gradient vanishes
to nothing at its earliest layers.
### Verified: residual connections prevent gradient collapse at depth
Two 30-layer stacks of identical blocks — one with a residual
connection, one without — measuring the gradient magnitude that
actually reaches the very first layer's input:
```python
import torch
import torch.nn as nn
d, depth = 32, 30
class BlockNoResidual(nn.Module):
def __init__(self):
super().__init__()
self.lin = nn.Linear(d, d)
def forward(self, x):
return torch.tanh(self.lin(x))
class BlockResidual(nn.Module):
def __init__(self):
super().__init__()
self.lin = nn.Linear(d, d)
def forward(self, x):
return x + torch.tanh(self.lin(x))
def measure_grad(block_cls):
blocks = nn.ModuleList([block_cls() for _ in range(depth)])
x = torch.randn(1, d, requires_grad=True)
h = x
for b in blocks:
h = b(h)
h.sum().backward()
return x.grad.abs().mean().item()
print(measure_grad(BlockNoResidual))
print(measure_grad(BlockResidual))
```
Verified output:
```
No residual, depth 30: 1.84e-08
With residual, depth 30: 4.73
```
**Roughly 8 orders of magnitude difference.** Without residual
connections, by the time a gradient signal has backpropagated through
30 stacked `tanh`-activated layers, it has shrunk to almost nothing —
the earliest layers would receive essentially no useful training
signal at all. With residual connections, the gradient at the same
depth stays a normal, usable magnitude. This is the same underlying
vanishing-gradient phenomenon track 00 module 04 covered for RNNs,
verified here for *depth* (many stacked layers) rather than *sequence
length* — residual connections are the transformer's specific fix for
this problem at depth.
### Why residual connections work: gradient has a direct path
The mechanism is simple once stated: `d(x + f(x))/dx = 1 + df(x)/dx`.
The `+1` term means a gradient can always flow straight through the `+
x` path, completely bypassing whatever `f` (attention or the FFN) does
— even if `f`'s own gradient shrinks toward zero (as `tanh`-activated
layers' gradients tend to, especially stacked many times), the `+1`
guarantees an unobstructed route back to earlier layers. This is
exactly why the gap above is so large: without the `+1` path, every
layer's gradient must pass entirely through `f`'s own gradient,
multiplying together and shrinking at every step.
### Layer normalization: forcing every token to a fixed scale
Layer normalization rescales each token's vector (across its own
feature dimensions — module 01, module 02) to have mean 0 and standard
deviation 1, regardless of the input's original scale:
```python
ln = nn.LayerNorm(8)
x = torch.randn(3, 8) * 5 + 10 # arbitrary scale and shift per row
out = ln(x)
print(x.mean(dim=-1), x.std(dim=-1))
print(out.mean(dim=-1), out.std(dim=-1, unbiased=False))
```
Verified output:
```
input means: [7.59, 11.75, 10.81] input stds: [4.91, 2.83, 6.78]
output means: [~0.0, ~0.0, ~0.0] output stds: [1.0, 1.0, 1.0]
```
No matter how differently-scaled the three input rows started (means
ranging from 7.6 to 11.8, standard deviations from 2.8 to 6.8), every
output row lands at (effectively) exactly mean 0, standard deviation 1.
This matters directly for the residual-connection story above: as
values pass through many stacked blocks, each adding its sub-layer's
output back onto an accumulating stream, that stream's scale could
otherwise drift arbitrarily large or small across dozens of layers —
layer normalization resets each token's scale at every step, keeping
values in a well-behaved range the way module 06 verified scaling was
needed to keep softmax well-behaved.
### Pre-norm vs. post-norm: where normalization sits relative to the residual
Two real, both-used conventions differ in exactly where layer
normalization is placed:
```
Post-norm (original 2017 paper): output = LayerNorm(x + Sublayer(x))
Pre-norm (most modern models): output = x + Sublayer(LayerNorm(x))
```
Pre-norm puts the raw, un-normalized `x` on the direct residual path
(the same `+1`-gradient path verified above), while post-norm applies
normalization *after* the addition, meaning that direct path is no
longer purely linear. This difference has a real, documented
consequence for training stability at large depth — pre-norm is the
more common choice in modern large models specifically because it
preserves cleaner gradient flow through the residual path, though it
isn't covered numerically in this module's exercises (a good candidate
for exercise 3 below).
## Reference
```
Concept What it does Verified
──────────────────── ─────────────────────────── ────────────────────
Residual connection output = x + Sublayer(x) gradient magnitude
at depth 30:
1.84e-08 (no
residual) vs. 4.73
(with residual)
Layer normalization normalize each token's own arbitrary input
features to mean 0, std 1 means/stds all
mapped to
~mean 0, std 1
```
```
Post-norm: LayerNorm(x + Sublayer(x)) — original paper
Pre-norm: x + Sublayer(LayerNorm(x)) — most modern large models
```
## Hands-on exercises
### Exercise 1 — reproduce the gradient-collapse measurement
Run the exact `measure_grad` comparison above at a few different
depths (try 5, 15, 30, 60) and confirm the gap between residual and
no-residual widens as depth increases — connecting the specific
mechanism (`+1` gradient term compounding across layers) to why the
effect gets worse, not just different, with more layers.
### Exercise 2 — reproduce the layer-normalization verification
Run the exact `nn.LayerNorm` example above with your own random input
values at a different scale and shift, and confirm the output always
lands at mean ~0, std ~1 per row regardless of the input's original
scale.
### Exercise 3 — measure the pre-norm vs. post-norm gradient difference
Build two versions of a deep stack (5-10 layers is enough to see an
effect) — one applying `LayerNorm` after the residual addition
(post-norm) and one applying it before the sub-layer, inside the
residual path (pre-norm) — and measure the gradient magnitude reaching
the input in each, similar to exercise 1's method. Connect what you
find to why pre-norm has become the more common choice for very deep
modern models.
## Independent challenge
A researcher proposes removing layer normalization entirely from a
transformer, arguing "residual connections already solve the
gradient-flow problem, so normalization is redundant." Using this
module's verified findings about what each component specifically
does, write two or three sentences on why this reasoning conflates two
different problems.
Discussion
Residual connections (verified above) solve a *gradient-magnitude*
problem — ensuring a gradient signal can reach early layers at depth.
Layer normalization solves a *different* problem — keeping each
token's own value scale bounded and consistent as it accumulates
contributions from many stacked sub-layers (verified: arbitrary input
scales all get mapped to a consistent mean-0/std-1 range). Removing
normalization wouldn't reintroduce the specific 8-order-of-magnitude
gradient collapse verified in exercise 1 (residual connections still
provide that `+1` gradient path), but it would let the *forward-pass*
values themselves drift to arbitrary, potentially very large or very
small scales across many layers, which independently causes training
instability (partly the same saturation risk module 06 verified for
unscaled attention scores, but here affecting every sub-layer's input
scale, not just attention specifically) — these are two separate
problems, each requiring its own fix.
Answers
1. `output = x + Sublayer(x)`. Verified: at depth 30, the gradient
magnitude reaching the input was ~1.84e-08 without residual
connections versus ~4.73 with them — roughly an 8-order-of-magnitude
difference.
2. `d(x + f(x))/dx = 1 + df(x)/dx` — the `+1` term guarantees a direct,
unobstructed gradient path back through every layer, regardless of
how small `f`'s own gradient becomes, preventing the multiplicative
shrinkage that causes gradient collapse in deep stacks.
3. That regardless of the input's original mean and standard deviation
(verified: means ranging 7.59-11.75, standard deviations ranging
2.83-6.78), the output always came out at effectively mean 0,
standard deviation 1 per row.
4. Post-norm applies `LayerNorm` after the residual addition
(`LayerNorm(x + Sublayer(x))`); pre-norm applies it before the
sub-layer, inside the residual path (`x + Sublayer(LayerNorm(x))`).
Pre-norm is more common in modern large models because it keeps the
direct residual path purely linear, preserving cleaner gradient flow
at large depth.
5. Because residual connections and layer normalization fix two
different problems — residual connections address gradient
*magnitude* at depth (verified above), while layer normalization
addresses per-token value *scale* consistency — removing
normalization would leave forward-pass value scales free to drift
across layers even though gradient flow itself would still be
preserved by the residual path.