Logit

A word’s raw, unnormalised score — the dot product of the model’s query vector with that word’s dictionary entry. One logit per vocabulary word.

Softmax

The function that turns a vector of logits into a probability distribution: exponentiate each, divide by the total. Bigger logits become bigger probabilities; the result sums to 1.

Unembedding

The matrix that maps the final vector to one logit per word. Its columns are word vectors. With tied weights it is the transpose of the embedding matrix — the same dictionary used for input lookup and output scoring.

Maximum-inner-product search (MIPS)

“Find the items whose vectors have the largest dot product with a query.” Ranking the vocabulary by logit is exactly this — a similarity search, with the dot product as the similarity.

Logit lens

A diagnostic: apply the unembedding to the residual at an intermediate layer to see what the model would predict if forced to commit there. It works because the residual lives, throughout the network, in the space the final readout uses.

Key-value memory (feed-forward block)

A way of reading the MLP/feed-forward sub-layers: each behaves like a stored key–value pair — it recognises a pattern in the residual (“Pietro called Paolo”) and writes an associated direction back (“toward Tarso”). The model’s per-token “lookups.”

Greedy decoding, sampling, temperature, top-k / top-p

Ways to pick a word from the ranked distribution. Greedy takes the top one. Sampling draws at random in proportion to probability. Temperature rescales the distribution before drawing — high = flatter/more varied, low = sharper/more predictable. Top-k / top-p restrict the draw to the most probable words.

RAG (retrieval-augmented generation)

Fetching relevant documents from an external store (via a vector-database similarity search) and inserting them into the prompt, so the model’s next-word search runs over a context grounded in real text.

MCP (Model Context Protocol)

A standard by which a model’s host application connects to external tools and data sources. In the picture of this note: a way for results computed outside the model to be written back onto the prompt “conveyor belt” as text the next-word loop then reads.

Token, vocabulary

A token is the unit of text the model reads — here a whole word; in production models usually a sub-word piece. The vocabulary is the fixed set of all possible tokens (28 in our dialogue game, 50,257 in GPT-2/GPT-3).

Embedding

The vector of real numbers that represents a token. The embedding matrix has one row per vocabulary token; “looking up” a token means reading its row. The rows are parameters — numbers learned during training — not values fetched from elsewhere (§1).

Encoder (retrieval / vector database)

A separately-trained model that turns a piece of text into one vector so that similar texts get nearby vectors. Used to fill a vector database for semantic search. Examples: OpenAI text-embedding-3, sentence-transformers, ColBERT. It is trained for relevance similarity, a different objective from next-token prediction (§1).

Gradient descent

The training procedure: nudge every parameter a little in the direction that reduces the model’s error, repeat millions of times. It is what “learns” the embeddings.

Residual stream

The running vector the Transformer keeps for each position and updates layer by layer; each attention/MLP block adds its output to it. The model’s prediction is read from the final residual. (See the QKV note.)

Attention head

A sub-mechanism inside a layer that, for each position, decides how much to read from every earlier position. Our toy has 2 layers × 4 heads. “Which head does what” is the central question of mechanistic interpretability.

Logits, softmax

The model’s raw output scores over the vocabulary are logits. Softmax turns them into a probability distribution (exponentiate, then normalise to sum 1).

Tied weights (embedding / unembedding)

Using the same matrix to map tokens→vectors (input) and vectors→token-scores (output, the unembedding ). Standard for small models; it means the “dictionary” the model searches at the end is the embedding matrix (§6).

Cosine similarity

A measure of how aligned two vectors are: the cosine of the angle between them (+1 = same direction, 0 = orthogonal, −1 = opposite). Ignores length, so it compares direction only.

PCA (Principal Component Analysis)

An unsupervised method that finds the directions along which the data spreads most (the principal components), computed from an eigen-decomposition / SVD of the centred data. Projecting onto the top two gives a 2-D map; the variance explained (e.g. “34.7%”) is the fraction of the data’s total spread those two directions capture — a built-in honesty meter (§4b).

Eigenvector / SVD

The linear-algebra machinery PCA runs on: the singular value decomposition (SVD) factorises the data matrix and hands back the principal directions and how much variance each carries. You do not need the details — just that one svd call yields the PCA axes.

Projection

Mapping high-dimensional points onto a lower-dimensional plane by taking dot products with chosen axes. PCA and concept-direction maps are true projections (the axes keep a fixed meaning); t-SNE is not (§4).

Gram-Schmidt / orthonormal

A recipe for turning two chosen direction vectors into a clean perpendicular (orthonormal) pair of axes, so the two coordinates you read off are independent. Used to build concept-direction planes (§4a, Appendix A).

Concept direction

An axis you choose to stand for a meaning — either a token’s own direction or a difference of tokens (e.g. Paolo − Pietro = “which leader”). Supervised: you bring the meaning (§4a).

t-SNE, UMAP, perplexity

t-SNE and UMAP are nonlinear methods that place points so that local neighbourhoods are preserved, producing cluster-y 2-D pictures. They are not projections: the axes, the distances between clusters, and the global layout carry no meaning, and the result changes with the random seed. Perplexity is t-SNE’s main knob, controlling roughly how many neighbours count as “local” (§4c).

Distributed representation / superposition

Distributed: a concept is carried by a pattern across many dimensions, not one. Superposition: a model packs more feature-directions into a space than it has dimensions, at angles chosen to minimise interference — which is why the axes are not individually meaningful (§2).

Linear representation hypothesis

The empirical idea that many high-level concepts correspond to straight-line directions in representation space — the reason concept-direction maps and analogies (king − man + woman) work at all, when they do (§4a, §7).

Sparse autoencoder (SAE)

A tool that decomposes a model’s activations into many sparse, often human-interpretable feature directions — used to read meaning out of superposition without dictating it in advance (§2, §7).

Ablation

Switching off one component (e.g. one attention head) and re-measuring behaviour, to test what it causally does — as opposed to what its attention pattern looks like. Fig 5’s “which write moved the residual” is the same spirit (§4a; the QKV note’s appendix).

A network you may already picture: the OCR convolutional net

Think of a classic optical-character-recognition (OCR) network — the kind that reads a handwritten digit or letter from a small image. Its workhorse is the convolution. The input is a grid of pixels. A small filter (say a patch of weights) slides across the image; at each location it multiplies the pixels under it by its weights and sums them into a single number. Sweeping the filter over the whole image produces a feature map — a new grid that lights up wherever the filter’s little pattern (an edge, a curve, a stroke) is present. A layer has many such filters, so it produces many feature maps. Stack a few convolution layers (with pooling in between to shrink the grid), and the network builds up from edges → strokes → loops → whole-character shapes.

It is worth stating the underlying operation precisely, because a clean version of it carries over to attention. A convolution is a weighted sum over a small local window of the input: at each position the filter computes a dot product between its fixed weights and the pixels it currently covers. (It is a weighted sum, not strictly an average — the weights need not sum to one and are often negative, which is how a filter can reward ink in one place and penalise it in another.) Two things are then layered on top. First, the same weights are reused at every position (weight sharing), so the operation is really one small pattern-detector swept across the whole image. Second, a layer stacks many such detectors and a later layer takes weighted combinations of their feature maps — and it is those combination weights, learned by training, that select which mixtures of low-level features best predict the correct character. So the intuition behind the original marker is right once it is split in two: each convolution is a local weighted sum, and the cross-feature mixing that “selects the useful combinations” happens when later layers combine feature maps. (Appendix B works a full convolution through by hand.)

At the very end the features are flattened and a softmax classifier turns them into a probability over the possible characters.

flowchart LR
    A["pixels<br/>(grid of ink/blank)"] -->|"filters slide<br/>shared 3×3 weights<br/>local receptive field,<br/>reused at every position"| B["feature maps<br/>one grid per filter:<br/>edges, strokes, ..."]
    B -->|"flatten +<br/>linear layer"| C["softmax over<br/>26 letters"]
    C --> D["a .02<br/>...<br/>e .91 ◄ answer<br/>..."]

Two properties make this work for images. First, each filter has a fixed, local receptive field: it only ever looks at a small neighborhood of pixels. Second, the same filter is reused at every position (weight sharing), so a stroke is recognized the same way wherever it sits on the page. Both properties are exactly right for images, because the clues needed to recognize a stroke are local and their meaning does not depend on where on the page they appear.

Why language breaks this, and why we need attention

Language does not have those two convenient properties. The piece of context a word depends on can be right next to it or hundreds of words back — and which earlier words matter depends on the content, not on a fixed offset. To resolve “it” you must find the noun it refers to, and that noun could be anywhere. A fixed, local filter cannot do this: it always looks in the same small window, regardless of what the sentence is about.

Attention is the fix. Instead of a fixed local window, an attention head computes, on the fly and for each token, how much to pull from every other token — and then pulls. The “receptive field” is no longer fixed by the architecture; it is decided at runtime, from the content, by the query–key matching we will detail in Section 2. In one sentence:

An attention head is the language analogue of a convolutional filter, except its receptive field is learned, dynamic, and content-dependent instead of fixed and local.

That single change — a receptive field the data gets to choose, every time — is what let neural networks finally handle long-range, content-driven dependencies, and it is the heart of the Transformer.

It is important not to over-claim, though: attention is the new idea, not the only idea. A Transformer interleaves two kinds of block. The attention heads are the novelty just described — the content-driven, dynamic receptive field that moves information between positions. Alongside them sit ordinary MLP (multi-layer perceptron) blocks, the same fully-connected, learned weighted-combination machinery that does the cross-feature mixing in a CNN — except here each MLP refines one token’s representation in place, with no reference to its neighbours. The two blocks divide the labour cleanly: attention is the only channel that lets tokens talk to each other, while the MLP is where each token’s representation is reshaped on its own. Sections 5 and 6 build both explicitly, and Appendix A shows what is lost if you switch the attention off and keep only the MLP.

What the output data actually is

It pays to be precise about what these networks produce, because the same description fits both the OCR net and the language model.

In both cases the network turns its input into a representation: a vector whose position in a high-dimensional space encodes the answer. In OCR, the final feature vector’s location says “this image sits in the region of the space that means the letter e.” In a Transformer, the inputs are pieces of text — tokens — and each token is represented by an embedding: a vector of numbers giving its coordinates in a high-dimensional “language space,” where direction and proximity stand in for meaning. The vector the model processes and maintains for each token — the residual stream, formally introduced in Section 4 — is best read as a modified embedding: it starts life as the raw token embedding and each layer nudges it to a new position that encodes everything the model has worked out about that token in its context.

So the output data are representations of entities, describing each entity’s most likely position in a solution space. And from that position the model reads out one of two things:

• a single crisp solution — the one best answer (take the highest-scoring class, i.e. the , equivalently temperature ); or
• a probability distribution over solutions — a graded set of plausible answers with weights.

The readout is the same machine in both networks: a linear projection followed by a softmax. OCR projects the final feature vector onto the alphabet and gets a distribution over letters; a language model projects the final token representation onto the vocabulary and gets a distribution over next tokens. Same structure, different solution space. (Section 5 shows exactly how the final representation’s direction encodes which tokens are likely and its magnitude encodes how confident the model is — the geometric version of “position in a solution space.”)

The logic of the processing: params convert inputs into entity representations

How does the input get from raw symbols to these context-aware representations? The recipe is the through-line of this whole note:

1. Look up. Each input token ID is replaced by its embedding — a first, context-free guess at its position in the space (a row of the embedding matrix ).
2. Read, transform, write — repeatedly. A stack of parameterized blocks then reads the current representations and writes adjustments back. Two kinds of block alternate:
   • Attention blocks move information between tokens (the dynamic receptive field above), letting each token’s representation absorb what it needs from the others.
   • MLP blocks refine each token’s representation in place, one position at a time (detailed in Section 6).
3. Read out. After such rounds (model layers) the representation is “finished,” and the unembedding projects it into the solution space to give the crisp answer or the distribution.

The parameters — in attention, in the MLP — are frozen after training. They are the learned knowledge: they encode the rule for how to move each representation, step by step, from a bare embedding toward its correct final position. Training is just the search for parameter values that put every entity in the right place in the solution space.

Why this framing matters for what follows. Because all the “thinking” lives in these incrementally-modified representations flowing through the residual stream, we can later ask very sharp questions about a trained model: where is a particular piece of behavior carried, and which block put it there? In small, fully-understood models one can even disable a single block and watch a specific behavior appear or vanish — the basis of the interpretability experiments these notes are meant to accompany (see Appendix A). The rest of this note builds the mechanism precisely enough to make those questions answerable.

Model parameters (frozen weights)

Weight	Shape	What it does
Token embedding		maps token IDs to vectors
Positional embedding (if any)		adds positional information
Per layer :		projects hidden → all heads’ Q
Per layer:		projects hidden → all heads’ K
Per layer:		projects hidden → all heads’ V
Per layer:		mixes head outputs back to residual
Per layer: MLP	,	non-linear transformation
Final LayerNorm		normalization scale/shift
Unembedding (often )		maps final hidden → logits

Total parameters scale roughly as (the famous result of Kaplan et al.).

Activations (recomputed every forward pass)

Activation	Shape	Notes
Input token IDs		integers in
Hidden state / residual stream		the running “meaning” per token, threaded through layers
, , per head	each, per head	computed from via , ,
Attention scores	per head	the routing pattern
Attention weights (post-softmax)	per head	row-stochastic, lower-triangular (causal mask)
Attention output (single head)		weighted sum of V rows
Attention output (all heads concat)		usually
Output of added to residual		written back into the residual stream
MLP output		also written back into the residual stream
Final hidden state (after layer , LayerNorm)		input to the unembedding
Logits		one row per token position
Logits of the last token		the one that matters for next-token prediction
Probability distribution over vocabulary	, sums to 1	of last-token logits

From last-token logits to the next-token distribution

After the final layer, the hidden state of the last position is multiplied by the unembedding matrix:

These are the logits: one real number per vocabulary token, unnormalized. Softmax converts them into probabilities:

where is the sampling temperature. At this becomes greedy (). At you get the raw model distribution. The sampler then draws the next token from this distribution, appends it to the sequence, and the loop continues.

KV cache size

The KV cache has total size:

The factor 2 is for K and V together. For a 7B model with , , , at FP16 (2 bytes), 32K context:

This is why long-context inference is memory-hungry, and why DeepSeek’s Multi-head Latent Attention (which compresses K and V into a low-rank latent space) is such a big deal — it can cut this by an order of magnitude.

Setup

• Vocabulary size . Say the vocabulary is {the, cat, dog, sat, ran, on, mat, floor, big, small, red, blue, fast, slow, a, and, ., is, was, .EOS} — 20 tokens indexed 0–19.
• Embedding dimension .
• One attention head, so .
• One layer (we’ll ignore MLPs for clarity).
• Context: 3 tokens. We’re going to compute attention for the sequence the cat sat, token IDs .

Step 1: token embedding

The embedding matrix is . After training it might look like (showing only the three rows we need):

E[0]  = [ 0.10, -0.20,  0.05,  0.40,  0.15]   "the"
E[1]  = [ 0.30,  0.50, -0.10,  0.20,  0.00]   "cat"
E[3]  = [-0.40,  0.10,  0.60, -0.30,  0.20]   "sat"

After looking up these three rows, the input to layer 1 is a matrix — three tokens, each as a 5-dimensional embedding:

X = [[ 0.10, -0.20,  0.05,  0.40,  0.15],
     [ 0.30,  0.50, -0.10,  0.20,  0.00],
     [-0.40,  0.10,  0.60, -0.30,  0.20]]

This is the initial residual stream , shape .

Step 2: compute Q, K, V

The weight matrices , , are each here.

General shape: for and for . In this toy example we have one head with , so the shape collapses to . In a real multi-head model the per-head slice has shape , not — the equality holds only after concatenating all heads.

is also called the IN matrix.

Let’s say:

W_Q = [[ 1.0,  0.0,  0.0,  0.5,  0.0],
       [ 0.0,  1.0,  0.0,  0.0,  0.5],
       [ 0.0,  0.0,  1.0,  0.0,  0.0],
       [ 0.5,  0.0,  0.0,  1.0,  0.0],
       [ 0.0,  0.5,  0.0,  0.0,  1.0]]

W_K = [[ 0.8,  0.2,  0.0,  0.0,  0.0],
       [ 0.2,  0.8,  0.0,  0.0,  0.0],
       [ 0.0,  0.0,  0.9,  0.1,  0.0],
       [ 0.0,  0.0,  0.1,  0.9,  0.0],
       [ 0.0,  0.0,  0.0,  0.0,  1.0]]

W_V = [[ 0.5,  0.5,  0.0,  0.0,  0.0],
       [ 0.5, -0.5,  0.0,  0.0,  0.0],
       [ 0.0,  0.0,  1.0,  0.0,  0.0],
       [ 0.0,  0.0,  0.0,  1.0,  0.0],
       [ 0.0,  0.0,  0.0,  0.0,  1.0]]

Compute , , . Each is in this single-head example. In general, per head, the shapes are for and , and for . In the three matrices below, each row is a token and each column indexes one coordinate of the per-head Q/K/V vector.

Q = [[ 0.30, -0.10,  0.05,  0.45,  0.05],   # for "the"
     [ 0.40,  0.50, -0.10,  0.35,  0.25],   # for "cat"
     [-0.25,  0.20,  0.60, -0.50,  0.25]]   # for "sat"

K = [[ 0.04, -0.14,  0.04,  0.41,  0.15],   # for "the"
     [ 0.34,  0.46, -0.10,  0.17,  0.00],   # for "cat"
     [-0.30, -0.00,  0.51, -0.21,  0.20]]   # for "sat"

V = [[-0.05,  0.15,  0.05,  0.40,  0.15],   # for "the"
     [ 0.40, -0.10, -0.10,  0.20,  0.00],   # for "cat"
     [-0.15, -0.25,  0.60, -0.30,  0.20]]   # for "sat"

(I’ve rounded these for readability; the principle is what matters.)

A fourth weight matrix, , (OUT matrix) is also a learned parameter of the attention block. Its job is to project the concatenated per-head attention outputs back into the residual stream’s dimension and (in multi-head attention) to mix information across heads.

General shape: . Here, with one head and , it collapses to . Note: is the concatenated head-output dimension, which in most architectures equals by design — that’s why usually looks like a square in implementations.

W_O = [[ 1.0,  0.0,  0.0,  0.0,  0.0],
       [ 0.0,  1.0,  0.0,  0.0,  0.0],
       [ 0.0,  0.0,  1.0,  0.0,  0.0],
       [ 0.0,  0.0,  0.0,  1.0,  0.0],
       [ 0.0,  0.0,  0.0,  0.0,  1.0]]

(For simplicity we’ve set ​ to the identity matrix here, meaning the attention output passes through unchanged. In a trained model ​​ would be a learned dense matrix that performs the head-mixing and projection described above.)

Step 3: attention scores

This is a matrix where entry is the dot product of token ’s query with token ’s key — a similarity score between tokens (not between heads), measuring how strongly token wants to attend to token .

QK^T = [[ 0.20,  0.07, -0.21],
        [-0.01,  0.40, -0.23],
        [-0.16,  0.06,  0.59]]

Now divide by :

QK^T / sqrt(5) = [[ 0.089,  0.031, -0.094],
                  [-0.004,  0.179, -0.103],
                  [-0.071,  0.027,  0.264]]

Step 4: causal mask + softmax

Since we’re doing autoregressive language modeling, each token can only attend to itself and earlier tokens. We mask the upper triangle to (so softmax gives them weight 0):

masked = [[ 0.089,  -inf,  -inf],
          [-0.004,  0.179,  -inf],
          [-0.071,  0.027,  0.264]]

Apply softmax row by row:

attention_weights = [[1.000, 0.000, 0.000],
                     [0.454, 0.546, 0.000],
                     [0.250, 0.276, 0.474]]

Read this carefully: the third row says that when computing the output for “sat”, the model attends 25% to “the”, 28% to “cat”, and 47% to itself. This is the attention pattern, the “who attends to whom” that QK produced.

Step 5: weighted sum of V

Multiply . Per head this is . With one head and , the result is :

attention_output = [[-0.050,  0.150,  0.050,  0.400,  0.150],   # "the" attends only to itself
                    [ 0.196,  0.014, -0.027,  0.291,  0.068],   # "cat" blends "the" + "cat"
                    [-0.022, -0.094,  0.262, -0.061,  0.132]]   # "sat" blends all three

The third row is the interesting one: it’s a weighted blend (25%/28%/47%) of the three V vectors. This is the V “payload” being routed along the attention edges that QK set up. The output for “sat” now incorporates information drawn from “the” and “cat” — that’s how the model learns long-range dependencies.

(With multiple heads, you’d have such blocks, one per head, computed in parallel from independent slices. They get concatenated along the last axis into a single tensor before the next step. Section 6 carries out exactly this multi-head case numerically.)

Step 6: through , back into the residual stream

The concatenated attention output (here just one head, so “concatenation” is a no-op) is projected by :

Shape arithmetic: . In our single-head toy, that’s .

This is the residual addition — the attention output is added to the previous residual stream, not replacing it. The hidden state is still shape .

In a real model, MLP layers would now run on top, also reading from and writing to the residual stream. We’d repeat the whole thing times. Here we have one layer, so we go straight to the output. (Section 6 adds the MLP and a second layer explicitly.)

Step 7: final hidden state → logits → next-token distribution

We only care about the next token, so we take the last row of : a single 5-vector, the final hidden state for “sat”. Suppose after and residual it’s:

h_last = [-0.351,  0.030,  0.802, -0.398,  0.247]

Multiply by the unembedding matrix , shape (one column per vocabulary token):

Result might look like:

logits = [-0.8, -0.2,  0.1,  0.5,  1.2,  2.4,  3.1,  2.8,  0.4, -0.1,
          -0.3,  0.0, -0.5, -0.4,  0.6,  0.2,  1.0, -0.2,  0.3, -1.5]
            the  cat  dog  sat  ran   on   mat floor  big small  red ...

The model thinks “on” (logit 2.4), “mat” (3.1), “floor” (2.8) are the most likely continuations of “the cat sat”. Applying softmax at temperature 1:

P("mat"   | "the cat sat") ≈ 0.32
P("floor" | "the cat sat") ≈ 0.24
P("on"    | "the cat sat") ≈ 0.16
P(others)                  ≈ remainder

This is the conditional distribution that the language model has been trained to approximate. The sampler picks a token from this — greedy would pick “mat” — appends it to the sequence, and the next forward pass begins.

Final step calculations in detail.

Setup

• .
• heads, so the per-head dimension is .
• layers, each one a full attention block + MLP block.
• MLP hidden size . (Real models use ; we shrink it so the matrices stay on the page.)
• ReLU nonlinearity in the MLP: , applied element-wise — it simply zeros out negatives.
• Context: the same 3 tokens, the cat sat, so .

Here is the whole journey as a shape table. Everything below just fills in the numbers for these rows.

Step	Operation	Output shape
token IDs	lookup
	embedding + positional
(all heads)	, etc.	each
per-head	slice into blocks	each, ×2
scores	per head	per head
weights	mask + softmax	per head
head output	weights	per head
concat	join 2 heads
attn write-back	concat
MLP pre-activation
MLP activation
MLP output
… repeat for layer 2 …
	final hidden state
last row	unembedding
softmax	distribution

Step 1: embedding + positional →

Each token’s 6-dimensional embedding, plus a positional embedding for its slot, gives the initial residual stream , shape :

h⁰ = [[ 0.10, -0.10,  0.05,  0.30,  0.15, -0.05],   # the  (pos 1)
      [ 0.40,  0.50, -0.20,  0.20,  0.10,  0.25],   # cat  (pos 2)
      [-0.50,  0.15,  0.65, -0.20,  0.15,  0.05]]   # sat  (pos 3)

Step 2: project to Q, K, V, then split into heads

is now (general shape ), and likewise . Multiplying gives a matrix. The crucial new idea: those 6 columns are two heads’ worth of Q stacked side by side — columns 1–3 are head 1’s query, columns 4–6 are head 2’s. The vertical bar marks the split:

Q = [[-0.24, -0.27, -0.43 | -0.05, -0.30, -0.01],   # the
     [-0.37,  0.29, -0.61 | -0.56,  0.17, -0.13],   # cat
     [ 0.20, -0.18, -0.00 |  0.37, -0.14,  0.09]]   # sat
       └──── head 1 ────┘   └──── head 2 ────┘

and are computed and split the same way. Each head now has its own query, key, and value. This is what “multi-head” means: one matrix multiply, then carve the result into independent lanes, each running the attention formula on its own slice.

Step 3: attention inside each head

Run Steps 3–5 of the previous example separately in each lane.

Head 1. Scaled scores , then causal mask and softmax:

scores₁ = [[-0.01,  -inf,  -inf],          weights₁ = [[1.00, 0.00, 0.00],
           [ 0.04,  0.07,  -inf],     →               [0.49, 0.51, 0.00],
           [-0.01, -0.04, -0.07]]                     [0.34, 0.33, 0.32]]

Weighted sum of head 1’s gives head 1’s output, shape :

head₁_out = [[-0.12, -0.04, -0.07],
             [-0.15,  0.40, -0.07],
             [-0.12,  0.20, -0.20]]

Head 2. Same procedure on head 2’s slice, giving its own pattern and output:

weights₂ = [[1.00, 0.00, 0.00],         head₂_out = [[ 0.06, -0.03,  0.01],
            [0.45, 0.55, 0.00],     →                [ 0.04, -0.06, -0.05],
            [0.36, 0.32, 0.32]]                      [-0.16,  0.11,  0.03]]

Notice the two heads produce different attention patterns from the same input — head 1 weights “cat” slightly more on row 2 (0.51), head 2 weights it more strongly (0.55). Each head is free to specialize. This is the structural fact the interpretability experiments hinge on: distinct heads can do distinct jobs, and you can study them one at a time.

Step 4: concatenate the heads

Glue the two head outputs back together, side by side, into one matrix:

concat = [[-0.12, -0.04, -0.07 |  0.06, -0.03,  0.01],
          [-0.15,  0.40, -0.07 |  0.04, -0.06, -0.05],
          [-0.12,  0.20, -0.20 | -0.16,  0.11,  0.03]]
            └─ head 1 out ──┘    └─ head 2 out ──┘

Step 5: and the residual addition

has shape . It mixes the two heads’ information together and maps it back to the residual width. The product is the attention block’s write-back, shape :

attn = [[-0.06,  0.10, -0.11,  0.05,  0.06,  0.01],
        [-0.03, -0.09, -0.10, -0.15,  0.30,  0.08],
        [ 0.19, -0.20,  0.23,  0.09,  0.34, -0.10]]

Add it to the residual stream — , still :

h_mid = [[ 0.04, -0.00, -0.06,  0.35,  0.21, -0.04],
         [ 0.37,  0.41, -0.30,  0.05,  0.40,  0.33],
         [-0.31, -0.05,  0.88, -0.11,  0.49, -0.05]]

Step 6: the MLP block

Now the piece the first example skipped. The MLP acts on the residual stream one token at a time — the same applied to every row independently, with no interaction between positions. (Hold onto that fact; the Appendix turns on it.)

First, expand from width 6 to width via (shape ): , shape :

pre = [[ 0.16, -0.07, -0.14,  0.10, -0.12, -0.05, -0.06, -0.29],
       [ 0.17, -0.38,  0.18,  0.24, -0.11, -0.26,  0.35, -0.24],
       [-0.02,  0.22,  0.51, -0.66, -0.29, -0.36, -0.45,  0.36]]

Apply ReLU — every negative entry becomes 0. This is the model’s only nonlinearity, and it’s what lets the MLP do more than a plain matrix multiply:

act = [[0.16, 0.00, 0.00, 0.10, 0.00, 0.00, 0.00, 0.00],
       [0.17, 0.00, 0.18, 0.24, 0.00, 0.00, 0.35, 0.00],
       [0.00, 0.22, 0.51, 0.00, 0.00, 0.00, 0.00, 0.36]]

Then contract back from width 8 to width 6 via (shape ): , shape :

mlp = [[ 0.07,  0.03, -0.04, -0.05,  0.08,  0.07],
       [ 0.08,  0.07, -0.22, -0.48,  0.24,  0.05],
       [ 0.79,  0.29, -0.27, -0.34,  0.26, -0.42]]

Add it back to the residual stream — , shape . Layer 1 is now complete:

h¹ = [[ 0.12,  0.03, -0.10,  0.31,  0.29,  0.04],
      [ 0.45,  0.48, -0.52, -0.43,  0.64,  0.38],
      [ 0.47,  0.24,  0.61, -0.45,  0.75, -0.47]]

Note the shape went inside the MLP: the residual stream stays width 6 everywhere; the width-8 expansion happens only inside the block and is contracted away before the write-back. The residual stream’s width is invariant — that constancy is what lets every block read and write the same tape.

Step 7: layer 2 (same shapes, new weights)

Layer 2 has its own , but the shapes and the procedure are identical to layer 1. It reads , runs two-head attention, writes back, runs its MLP, writes back, and produces . We show only the write-backs and the result:

attn² (write-back)   h_mid² = h¹ + attn²        mlp²                 h² = h_mid² + mlp²
[[-0.09,-0.27, 0.01,  [[ 0.02,-0.24,-0.09,       [[-0.04,-0.27,-0.37,  [[-0.02,-0.51,-0.46,
  -0.10,-0.02,-0.08],    0.21, 0.27,-0.05],         -0.66, 0.06, 0.18],   -0.46, 0.33, 0.14],
 [-0.20,-0.48, 0.08,   [ 0.25, 0.01,-0.44,        [-0.17,-0.40,-0.58,   [ 0.08,-0.39,-1.01,
   0.12,-0.06,-0.39],    -0.31, 0.58,-0.02],        -0.93, 0.17, 0.33],   -1.24, 0.75, 0.32],
 [-0.46,-0.16, 0.31,   [ 0.02, 0.08, 0.92,        [-0.22,-0.53,-1.20,   [-0.21,-0.45,-0.28,
   0.21,-0.28,-0.47]]    -0.24, 0.47,-0.94]]        -2.21, 0.90, 0.26]]   -2.45, 1.37,-0.67]]

Each of these is . After layers the residual stream is still — the same shape it started as. Every layer reshaped the contents, never the shape.

Step 8: read out the last token

Exactly as in Section 5. Take the last row of (the representation for “sat”, now informed by two full layers of attention + MLP):

h²_last = [-0.21, -0.45, -0.28, -2.45,  1.37, -0.67]

Multiply by the unembedding (shape ) to get logits, then softmax. Showing the first 10 vocabulary columns:

logits = [-0.48,  0.27,  2.01, -0.91,  0.35, -1.04, -1.80,  1.03,  0.60,  0.36]
probs  = [ 0.04,  0.07,  0.42,  0.02,  0.08,  0.02,  0.01,  0.16,  0.10,  0.08]

(With random, untrained weights the specific winner carries no meaning — what matters is that the pipeline produced a clean -shaped distribution that sums to 1.)

What this example added

Reading the shape table top to bottom, three things are now visible that the single-head example could not show:

1. Heads are lanes. One multiply produces all heads at once; the result is sliced into independent blocks, each running attention alone, then concatenated and remixed by . The width bookkeeping is , and here exactly.
2. The MLP is a per-token refinery. It expands each token’s vector to , applies ReLU, contracts back, and adds the result to the residual — touching each position in isolation. The width bookkeeping is .
3. Layers stack without changing shape. The residual stream is a tape that every block reads from and writes to additively. Stacking layers just repeats the read–transform–write cycle; the shape is invariant from to .

The baseline: a network with only MLP blocks

Recall from Section 6 that an MLP block processes the residual stream one token at a time: the same , ReLU, applied to each row independently, with no reference to any other position. So a network built only from MLP blocks processes every token in complete isolation. It can learn a fixed mapping “this token (at this position) → that output,” i.e. per-token and position-conditioned statistics — but it has no mechanism for one token to influence another’s representation. Whatever “sat” becomes, it becomes without ever consulting “the” or “cat.”

Method 1: remove the attention layers entirely

Delete the attention sub-blocks and wire the embeddings straight into the MLP stack. The update rule becomes simply

This is the pure-MLP baseline. In the running example, the representation of “sat” can now never absorb anything from “the” or “cat” — each column of the residual stream is processed down its own private pipe. A relational rule is therefore impossible: anything that requires comparing two positions — for instance “the next item must differ from the current one” — cannot be expressed, because the two positions never meet.

Method 2: keep the architecture, make attention transparent

The second way disables attention without deleting anything. Keep the full Transformer architecture — all the machinery — but fabricate the weights so that the attention block writes nothing into the residual stream.

Recall the write-back from Section 6: the attention block’s contribution is , and it is added to the residual. So set

Now, no matter what attention pattern computes, the value added to the residual is the zero vector:

The residual stream passes through the attention block untouched. The Q/K/V machinery still runs, still computes attention patterns — but those patterns are transparent: they have no effect on anything downstream. Functionally, the model is once again the pure-MLP network of Method 1.

(You might ask whether there’s a subtler “identity” fabrication — make each token attend only to itself, so attention copies each value through. That doesn’t give transparency: copying each token’s value and adding it back doubles the contribution rather than leaving the stream unchanged. True transparency means the block adds zero, which is what zeroing the write-back achieves.)

Both roads lead to the same place

Whether you disable attention by deletion (Method 1) or by making it transparent (Method 2), the Transformer collapses to a per-position MLP network — a stack that refines each token in isolation and can never move information between positions. This is the precise sense in which:

Attention is the only channel through which tokens communicate. Everything relational a language model does must be carried by attention, because it is the only operation that moves information across positions.

That is also why the tiny-model story these notes accompany is about attention heads specifically: if a rule relates one token to another, the circuit that enforces it has to live in the attention machinery — there is nowhere else for it to be.

Why Method 2 is the important one: ablation

Method 2 has a feature deletion lacks: it is selective. The write-back matrix is organized in blocks — one slice of rows per head (Section 6, Step 5). Zero out just one head’s slice, and you make exactly that head transparent while leaving every other head working. Re-run the model, measure what changes, and you have a causal probe: what does this one head actually do?

This per-head version of Method 2 is called ablation, and it is the workhorse of mechanistic interpretability. Two findings from tiny, fully-understood models show why it matters:

• A rule can rest on a single head. Ablate that one head and a behavior the model performed perfectly collapses; ablate any other head and nothing changes. The behavior was carried, causally, by one specific lane in one specific layer.
• Attention patterns can mislead. A head whose attention looks like it implements a rule — say it stares almost entirely at the relevant earlier token — may, when ablated, turn out to change nothing: it was not load-bearing. Conversely a head with a messy, unremarkable-looking pattern may be the one holding the rule. The attention pattern tells you what a head looks at; only ablation tells you what it does.

That second lesson is worth underlining, because it is exactly where intuition goes wrong: you cannot read a head’s function off its attention picture. You have to switch the head off — Method 2, one head at a time — and watch what breaks. Disabling attention, far from being a destructive curiosity, is therefore the single most useful tool for finding out where in a network a behavior lives — which is the question the accompanying tiny-language-model experiments are built to answer.

The image

Take a tiny grayscale image. Each pixel is (blank) or (ink). This one shows a vertical stroke down the middle column — the kind of mark that distinguishes, say, a 1 or the spine of a T:

       col: 0 1 2 3 4
row 0:      0 0 1 0 0
row 1:      0 0 1 0 0
row 2:      0 0 1 0 0
row 3:      0 0 1 0 0
row 4:      0 0 1 0 0

One filter, sliding

A filter is a small fixed grid of weights. Here is a vertical-stroke detector: it rewards ink in its center column and punishes ink on either side, so it responds most strongly to a vertical line.

F_vert = [[-1,  2, -1],
          [-1,  2, -1],
          [-1,  2, -1]]

To convolve, we slide this filter over every window of the image; at each stop we multiply overlapping cells and sum to a single number. A image with a filter has valid stops, so the output (the feature map) is .

Look at two stops to see the mechanism:

Top-left window (rows 0–2, cols 0–2) — the stroke is off to the right, so the filter’s center column sits on blanks:

window      = [[0,0,1],     elementwise·F_vert, summed:
               [0,0,1],     each row: 0·(-1) + 0·(2) + 1·(-1) = -1
               [0,0,1]]     three rows → -3

Top-center window (rows 0–2, cols 1–3) — now the stroke lines up under the filter’s center column:

window      = [[0,1,0],     each row: 0·(-1) + 1·(2) + 0·(-1) = +2
               [0,1,0],     three rows → +6
               [0,1,0]]

Do this at all nine stops and you get the feature map. The center column lights up (+6); the flanks are suppressed (−3):

conv = [[-3,  6, -3],
        [-3,  6, -3],
        [-3,  6, -3]]

ReLU, then pooling

Apply ReLU (zero out negatives) — the map keeps only the positive evidence “a vertical stroke is here”:

relu(conv) = [[0, 6, 0],
              [0, 6, 0],
              [0, 6, 0]]

Then max-pool with a window to shrink the grid and add a little position tolerance (each output cell is the max of a patch). The map becomes :

pool = [[6, 6],
        [6, 6]]

The pooled map is uniformly high: this filter is shouting “vertical stroke, present.” Pooling means it would keep shouting even if the stroke shifted a pixel — the detector is now slightly position-invariant, exactly the property Section 1 said convolution buys you.

A layer has many filters

A real layer applies many filters in parallel, each producing its own feature map. Add a second filter, a horizontal-stroke detector (ink rewarded in the center row):

F_horiz = [[-1, -1, -1],
           [ 2,  2,  2],
           [-1, -1, -1]]

Run it on the same vertical-stroke image and every window cancels to zero — there is no horizontal ink to reward:

conv = relu = pool = all zeros

So the two filters disagree, informatively: the vertical detector fires, the horizontal detector is silent. That contrast is the feature the classifier wants.

Flatten and classify

Flatten the two pooled maps into one feature vector (four numbers per filter, eight in all):

features = [6, 6, 6, 6,   0, 0, 0, 0]
            └ vertical ┘   └ horizontal ┘

A final linear layer reads these features into one score per class — let the “vertical” class sum the vertical-filter features and the “horizontal” class sum the horizontal ones — and softmax turns the scores into probabilities:

logits = [24,  0]            # vertical, horizontal
softmax ≈ [1.00, 0.00]       # "this is a vertical stroke"

The network has converted a grid of pixels, step by step, into a point in a tiny two-class solution space, and read out a crisp answer — the same arc as the language model, just over characters instead of next tokens. (The probability is emphatic because this is a noise-free toy; on real handwriting the distribution would be softer.)

Filter vs head, side by side

Now the comparison that motivated this appendix. A filter and a head are both feature detectors that get reused across positions, but they differ in the one respect that matters for language:

	Convolutional filter (CNN)	Attention head (Transformer)
What it stores	a fixed pattern of weights	three projections (and shares )
Receptive field	fixed and local — always the same small window	dynamic and global — chosen at runtime, can reach any earlier token
How it matches	slides the same pattern over every position; fires where the pixels match it	computes, from the content, a query–key similarity to decide which positions to read
What “reuse across positions” means	weight sharing: identical weights at every location	the same at every position, but the attention pattern is recomputed per input
Output at a position	one activation = how well the local patch matches the pattern	a weighted blend of other positions’ V payloads
Set by the data…	at training time (the filter weights are learned, then frozen)	at training time and at run time (the pattern depends on the actual tokens)

The crucial row is receptive field. The vertical filter above can only ever see a patch; to relate ink in the top-left corner to ink in the bottom-right, a CNN must stack many layers until their windows overlap. That is fine for images, where the clues are local. A head pays no such toll: the query for one token can match the key of a token hundreds of positions away in a single step, and which token it matches is decided by the content, not fixed by the architecture. That is the whole reason attention displaced convolution for language — and, looping back to Appendix A, it is also why a relational rule in a language model lives in a head: the head is the only component whose reach is wide enough, and content-driven enough, to relate one token to another.

Token, vocabulary

A token is the unit of text the model reads — a whole word in our toy examples, usually a sub-word piece in production models. The vocabulary () is the fixed set of all possible tokens (20 in Section 5’s toy, 50,257 in GPT-2).

Embedding, embedding matrix

The vector of real numbers that represents a token — its coordinates in the model’s high-dimensional “language space.” The embedding matrix has one row per vocabulary token; “looking up” a token means reading its row.

Residual stream

The running vector the Transformer maintains for each position and updates layer by layer; every attention and MLP block adds its output to it. It is the only place the model’s “thinking” lives, and the prediction is read from its final state.

Attention head, Q / K / V

A sub-mechanism that, for each token, decides how much to read from every earlier token. It does so with three learned projections of the residual: the query (“what am I looking for?”), the key (“what do I offer to match against?”), and the value (“what payload do I carry if matched?”). Query–key similarity sets the attention pattern; the values are what flow along it.

Multi-head attention

Running attention heads in parallel on independent slices of Q/K/V, then concatenating their outputs and mixing them with . Each head can specialise in a different relation.

The four learned weight matrices of an attention block: three that project the residual into queries, keys and values, and that maps the concatenated head outputs back into the residual stream.

MLP block (, ReLU)

The fully-connected “feed-forward” sub-layer that refines each token’s vector in place. It expands the vector to a wider hidden size via , applies a non-linearity (here ReLU, which zeroes negatives), and contracts back via . It never mixes information across positions.

Logits, softmax, temperature

The model’s raw, unnormalised scores over the vocabulary are logits. Softmax turns them into a probability distribution (exponentiate, then normalise to sum to one). Temperature rescales the logits before softmax: low sharpens the distribution (greedy at ), high flattens it.

Unembedding , weight tying

The matrix that maps the final residual vector to one logit per vocabulary token; each column is a token’s representation. Weight tying sets — the same dictionary used for input lookup and output scoring (standard in GPT-2, Llama, and our toys).

LayerNorm / RMSNorm

A normalisation applied to the residual vector (notably just before the unembedding) that rescales its components to a controlled magnitude and applies a learned per-dimension scale.

Causal mask

The rule that each token may attend only to itself and earlier tokens. Implemented by setting the upper triangle of the attention scores to before softmax, so future positions get weight zero.

KV cache

The stored keys and values of all past tokens, reused at each generation step so the new token’s query can attend back to the whole history. Queries are not cached — hence “KV”, not “QKV”.

Prefill vs. generation

Prefill processes the whole prompt at once, writing every token’s K and V into the cache. Generation then adds one token at a time, growing the cache by one row of K and V per step.

Ablation

Switching off one component — e.g. zeroing one head’s slice of — and re-measuring behaviour, to test what that component causally does (Appendix A).

Convolution, filter, feature map (CNN terms)

A convolution slides a small fixed filter of weights over an image, computing a local weighted sum at each position; the resulting grid is a feature map. The contrast with an attention head — fixed local window vs. dynamic content-driven reach — motivates Section 1 and Appendix B.