Large Language Models on Bitsy Services Wiki

Conventions

Mon, 01 Jan 0001 00:00:00 +0000

Every diagram and sentence here uses one layout, declared once so no page has to re-orient you.

Depth runs left to right. Embedding at the left edge, block 0 next, block 11 further right, unembedding at the right edge. A vector that has been through more blocks is further right. Never “deeper,” never “up the stack.”
Sequence runs top to bottom. One token position is one row: first token on top, the token you’re predicting from at the bottom.
The feature axis is row width. A row is d_model numbers wide, and the only thing that changes that is the MLP bulge — which changes it straight back.

                        depth  →

  pos 0  "The"   [···········]──[···········]──▶ logits
  pos 1  " cat"  [···········]──[···········]──▶ logits
  pos 2  " sat"  [···········]──[···········]──▶ logits
    │
    ▼             embed          block 0 … 11
 sequence        └── d_model ──┘

What the layout buys you#

Attention moves information between rows — a row reads the rows above it. The MLP moves it within a row, never looking at another. Here that’s literally vertical versus horizontal: the cleanest one-sentence account of a block you’ll get.

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

The words this subsection uses — and it uses only these. Third column: what the same thing is called elsewhere, so you can translate when you leave.

Term	Means	Elsewhere called
row	one token position’s vector at some depth — `d_model` numbers wide	activation, hidden state, token representation
`d_model`	the row width. 768 in GPT-2 small, and fixed from embedding to unembedding	`n_embd`, hidden size, model dimension
residual stream	the row seen as a bus running rightward through every block: blocks add into it and never overwrite it	hidden state, skip path
block	one attention plus one MLP, each wrapped in a norm and a skip connection. GPT-2 small has 12	layer, transformer layer, decoder layer
MLP bulge	the MLP widening a row to 4×`d_model` (3072) and back down. The only place a row isn’t `d_model` wide	feed-forward, FFN, hidden dim
head	one attention channel: its own Q/K/V projections into a 64-wide subspace. 12 per block in GPT-2 small	attention head
attention pattern	one row’s post-softmax weights over the rows it can see. Sums to 1	attention weights, attention map
KV cache	keys and values already computed for earlier rows, kept so the next token doesn’t recompute them	`past_key_values`, decoder cache
logits	the raw scores at the right edge — one per vocabulary entry (50,257), before softmax	scores, unnormalized log-probs
unembedding	the matrix that turns the final row into logits. Tied to the embedding in GPT-2	`lm_head`, output head, `W_U`
perplexity	`exp(loss)` — roughly, how many tokens the model is choosing between	PPL
feature	a direction in the stream that means something	concept, latent

Words this subsection avoids#

“Layer” — ambiguous between a block and a sublayer inside it. Say block, attention, or MLP. (LayerNorm keeps its name; it’s a proper noun.)
“Deeper,” “up the stack” — spatially wrong here; depth runs rightward. See conventions.
“Column” — a row is never one, whatever shape the tensor is in memory.

Depends on / leads to#

Depends on conventions. Leads to every other page in the subsection.

GPT-2

Mon, 01 Jan 0001 00:00:00 +0000

GPT-2 is the language model this section takes apart. OpenAI built it to test one idea — that a program trained only to guess the next piece of text, given enough ordinary writing to practise on, would pick up skills nobody set out to teach it — and it worked well enough to point the whole field in the direction it has been going ever since. It is also, by current standards, small and completely open: the weights are published, it runs on a laptop, and it is built the same way as the models people pay for today. Pivotal and openable at once is a rare combination, and it is why every page here uses GPT-2 as its worked example.

Tokenization

Mon, 01 Jan 0001 00:00:00 +0000

A model never sees text. It sees a list of integers, and the tokenizer is the only component that ever touched a character. GPT-2’s is byte-pair encoding: start from raw bytes, repeatedly merge the most frequent adjacent pair, keep the merge table. Encoding replays those merges in the order they were learned — not longest-match, which is the usual misreading. The vocabulary is what they produced: 50,000 merges plus the 256 raw bytes and an end-of-text marker, so 50,257 entries.

Embeddings

Mon, 01 Jan 0001 00:00:00 +0000

An embedding is where a token stops being an arbitrary label and becomes a position in a space the model has opinions about. Every entry in the vocabulary owns its own list of numbers, learned during training, and embedding a token means fetching its list — pulling one row out of a table. Nothing is calculated. This is the model’s first move, and it is the least clever thing it does all day.

The Residual Stream

Mon, 01 Jan 0001 00:00:00 +0000

Nothing in a transformer ever replaces a token’s vector. Each part of the model reads it, works something out, and adds the result on top of what’s already there. What comes out the far end is the starting vector plus every contribution made along the way — a running total. That total is the residual stream, and it is the only thing travelling through the model at all.

Why a sum, and not a pipeline#

Picture the obvious design first, because almost everyone does. Each stage takes the previous stage’s output, transforms it, and hands on a replacement — a relay race, one baton, passed along. That’s how a plain deep network works.

Attention

Mon, 01 Jan 0001 00:00:00 +0000

Attention is the part of a model that lets one word take account of another. Everything else in a transformer handles each position on its own: a word is processed without reference to its neighbours, as though it arrived alone. Attention is the exception, and it is the only exception — the sole mechanism by which information moves between positions. Remove it and what’s left is a machine that reads every word in isolation and can never relate any of them.

One Attention Head

Mon, 01 Jan 0001 00:00:00 +0000

Attention as an idea is a row asking the rows above it a question and blending the answers. This page is that idea as arithmetic. One head is a single complete instance of the machinery — a block runs twelve of them side by side, but one shows the entire operation.

Here it is, from a single row’s point of view. The row emits a query: what am I looking for? Every row at or above it exposes a key: what I’ve got. Dot the query against each key and you get a score per row — how much this row wants that one. Softmax those scores into an attention pattern: weights that sum to 1 across the rows it’s allowed to see. Then take those rows’ value vectors, weight them by the pattern, sum, and map the result back to d_model. That’s what the head adds to the row’s residual stream.

Q/K/V as Three Projections

Mon, 01 Jan 0001 00:00:00 +0000

Attention talks about a row’s query, its key, and its value as though the row were carrying three things around with it. It isn’t. There is only the row. Query, key, and value are three different ways of reading it — three learned lenses, each trained to pull a different aspect out of the same numbers. Nothing is stored. All three are worked out from the row on the spot, every time attention runs.

The Causal Mask

Mon, 01 Jan 0001 00:00:00 +0000

A row may not read the rows below it. Before the softmax, every score pointing at a row further down the sequence is set to −∞, so its weight comes out exactly 0. That’s the mechanism: a triangle of −∞ added to the score matrix.

The reason is training. One forward pass over a 1024-token sequence makes a prediction at every row — 1023 of which have a next token to score against. If row 5 could see row 6, the answer to “what follows row 5?” would be sitting in the input, the loss would go to zero, and the model would learn nothing. The mask is what makes training on every position at once honest.

Multi-Head Attention

Mon, 01 Jan 0001 00:00:00 +0000

One head is a narrow channel: a 64-wide query chasing one kind of relationship. GPT-2 small runs 12 of them per block, in parallel, over the same row, and they never talk to each other.

The row doesn’t get wider. d_model is split, not multiplied: 12 heads × 64 = 768. Each head reads the full row through its own Q/K/V projections, produces its own 64-wide result, and the twelve results are concatenated back to 768, passed through one output matrix, and added to the stream as a single write. Twelve independent reads, one write.

The MLP

Mon, 01 Jan 0001 00:00:00 +0000

Attention moves information between rows. The MLP is what the block does to a row once it has what it needs — and it never looks at another row.

Mechanically it’s two matrices with a nonlinearity between them: 768 → 3072, GELU, 3072 → 768. That widening is the MLP bulge, the only place in the model where a row isn’t d_model wide. Nothing mixes across rows on the way through; each row goes in and comes back alone.

GELU and SwiGLU

Mon, 01 Jan 0001 00:00:00 +0000

Almost everything a transformer does to a row is, arithmetically, a straight line: multiply by a matrix, add a vector, multiply by another matrix. Straight lines have an inconvenient property — chain as many as you like and the result is just another straight line. A model built only from them would be no more capable with a hundred blocks than with one, however many parameters you poured in. The activation function is the small bend in the middle of the MLP that breaks the straightness, and it is the reason depth buys anything at all. GPT-2 bends with GELU; nearly everything since bends with SwiGLU.

LayerNorm and RMSNorm

Mon, 01 Jan 0001 00:00:00 +0000

Before attention or the MLP reads the row, the row is normalized: subtract the mean of its 768 numbers, divide by their standard deviation, then scale and shift by learned per-feature parameters. That’s LayerNorm. It works within a row, along the feature axis — never across rows, so it can’t leak anything between token positions. (The name is historical; it normalizes a row, not a block.)

GPT-2 is pre-norm: row = row + attention(norm(row)). Read that placement carefully. The norm applies to the copy attention reads. It does not apply to the stream. The residual stream is never normalized in place — it keeps whatever magnitude it has accumulated, and it grows rightward, hard. In GPT-2 small the last row’s norm runs from about 5 at the embedding to about 400 leaving block 11: a factor of a hundred. Exactly once, at the right edge, a final norm rescales it before the unembedding reads it.

Skip Connections

Mon, 01 Jan 0001 00:00:00 +0000

The + in row = row + attention(norm(row)) is the skip connection, and it’s the reason the residual stream exists. The stream is the accumulated skip path.

It does two jobs.

It makes “do nothing” the default. A block whose attention and MLP output zeros is the identity, so a new block starts out harmless and learns a correction instead of having to earn back what its predecessors built. That is the argument for the residual stream and it’s made there; the + is simply the thing that does it.

The Unembedding and Logits

Mon, 01 Jan 0001 00:00:00 +0000

The final row is 768 numbers. The vocabulary has 50,257 entries. The unembedding closes that gap with a single matrix multiply and nothing else.

After the final norm, the row is dotted against one vector per vocabulary entry. The score for token t is dot(row, W_U[t]) — literally: how much does this row point in token t’s direction? Those 50,257 scores are the logits. No search, no nonlinearity, no table of stored continuations. The model’s answer is one vector, and the vocabulary is a set of directions to compare it against — direction picks the winner, magnitude decides how confident the logits look.

Softmax and Temperature

Mon, 01 Jan 0001 00:00:00 +0000

Logits are scores: unbounded, signed, meaningless alone. Softmax turns them into probabilities — exponentiate, normalize — with a temperature knob in the middle:

`1`	`p_i = exp(z_i / T) / Σ_j exp(z_j / T)`

Two properties are worth knowing cold.

Only differences matter. Add a constant to every logit and nothing changes; the constant cancels top and bottom. A logit’s absolute value tells you nothing. The gap to the next logit tells you everything.

Sampling Strategies

Mon, 01 Jan 0001 00:00:00 +0000

Softmax hands you a distribution over 50,257 tokens. Sampling turns it into one token — the only real choice you get at inference, and the one most likely to be blamed on the model.

Greedy takes the argmax. Deterministic, and it degenerates: fluent text that falls into loops, because the most likely continuation of a phrase you’ve just said is often to say it again.

Pure sampling draws from the whole distribution. It never loops, and it wanders, because the tail is enormous. After “the cat sat on the,” the ~49,500 tokens below p = 0.0001 carry more probability between them (0.10) than the single best token does (0.08). Sample long enough and you will pick from the garbage.

The Loss Function

Mon, 01 Jan 0001 00:00:00 +0000

One number trains the entire model: the negative log-probability the model assigned to the token that actually came next, averaged over every row.

`1`	`loss = −(1/n) Σ log p(actual next token \| rows above)`

That’s cross-entropy, and it’s the only signal. Nobody tells the model what a verb is, that Paris is in France, or which of two continuations reads better. It’s told, at each of a trillion positions, how much probability it put on the token that turned up.

Perplexity

Mon, 01 Jan 0001 00:00:00 +0000

Perplexity is the standard single number for how good a language model is at its one core job: predicting the next token. It restates the model’s average surprise as an effective number of choices — how many equally-likely options the model is, in effect, deciding between at each step. Fewer is better; a model that always knew exactly what came next would have nothing left to choose between. It’s the figure papers quote to claim one model fits a body of text better than another, though the same model scores differently depending only on how that text was cut into tokens — a caveat that turns out to be most of the story of reading the number honestly.

Backprop Through One Weight

Mon, 01 Jan 0001 00:00:00 +0000

Pick one number out of GPT-2 small’s 124 million — a single entry w in block 6’s MLP input matrix. Backprop answers exactly one question about it: if I nudge w up a hair, does the loss go up or down, and how fast? That’s ∂L/∂w. Training is that answer, applied to every weight, a few hundred thousand times.

The chain rule gets it there in one sweep. The loss is computed at the right edge; its gradient flows right to left, and at each operation the gradient arriving from the right is multiplied by that operation’s local derivative. By the time it reaches w, the number is a sum over every path from w to the loss — through the skip connections, through every block to its right — each path a product of local derivatives. It’s a sum, not a product. Think “product” and you’ve reinvented the vanishing gradient that skips exist to prevent.

Weight Sharing Across Positions

Mon, 01 Jan 0001 00:00:00 +0000

There is no “block 6 for position 300.” Every row runs through the same weights: one set of Q/K/V matrices, one MLP, per block, applied identically at position 1 and position 900.

This isn’t a memory optimization. It’s what makes the thing a language model instead of a lookup table indexed by position. Whatever “attend to the subject of this clause” costs to learn, it’s learned once and works everywhere — and gradients from every position in every sequence pile into the same matrices, which is why one sequence teaches a weight a thousand times over.

The KV Cache

Mon, 01 Jan 0001 00:00:00 +0000

Generating token 500 means running a forward pass over 499 rows to produce one new row. Do that naively for every token and you rebuild the entire prefix each time — the same keys and values, from the same unchanged rows, again and again.

The causal mask says you don’t have to. A row can only depend on itself and the rows above it, so once a row is computed it’s final: nothing you append below can change it, and its key and value are constants. Keep them.

Grouped-Query Attention

Mon, 01 Jan 0001 00:00:00 +0000

When a model is served to real users, the thing that fills the graphics card is not always the model. It’s the pile of remembered arithmetic kept for every token of every conversation in flight — and multi-head attention makes that pile as large as it can possibly be, by giving every head its own private copy. Grouped-query attention is the observation that they don’t all need one. Put the heads in groups, let a group share, and most of the memory goes away while the model carries on asking just as many questions as before. It costs a little quality, and it is why essentially every model served at scale today is built this way.

Positional Encoding

Mon, 01 Jan 0001 00:00:00 +0000

A query dotted with a key doesn’t know where either row sits. Attention, by itself, is a bag of rows. Position has to be injected on purpose.

GPT-2 injects it once, at the left edge. It learns a vector per position — wpe, one row per slot — and adds it to the embedding before block 0 sees anything. Same width, added rather than concatenated, so position costs the row no capacity; it just perturbs it. Every block and every head downstream reads position out of a signal mixed into the row’s content at the start.

RoPE

Mon, 01 Jan 0001 00:00:00 +0000

Attention scores a query against a key with a dot product, and a dot product has no idea where either token sits in the sequence. Order has to be put in on purpose. Rotary position embedding (RoPE) puts it in as rotation — and does it so that the score between two tokens depends only on the distance between them.

Position is invisible to a dot product#

Attention works by dotting a query row against a key row: a large dot product means “this token is relevant to that one.” But the dot product looks only at the contents of the two vectors. It cannot tell whether the key is the token right before the query or five hundred tokens back. Swap two rows of the sequence and every attention score comes out identical. Order isn’t in the arithmetic, so it has to be injected — which is the job of positional encoding.

Training vs Inference Parallelism

Mon, 01 Jan 0001 00:00:00 +0000

The same weights, run two completely different ways.

Training does every row at once. One forward pass over 1024 tokens computes all 1024 rows in parallel and scores 1023 next-token predictions — the causal mask is what makes that legal. The GPU is doing enormous matmuls, it’s compute-bound, and that’s why training is a throughput problem you solve with more FLOPs.

Inference can’t. Token 501 depends on token 500, which you have to sample first. Generation is strictly serial: one row, one forward pass, repeat. And with a KV cache the arithmetic per row is trivial — you push a single 768-wide row through 124M weights and get about one multiply out of each. Every weight is read; none is reused. The parallel pass reads exactly the same weights and reuses each one a thousand times over. That difference — arithmetic intensity, not FLOPs — is what makes decode memory-bandwidth-bound.

Context Length and the O(n²) Cost

Mon, 01 Jan 0001 00:00:00 +0000

Every row scores every row it can see: an n × n matrix of scores, per head, per block — the only thing in a transformer that grows quadratically with the sequence.

Everything else is linear: projections, MLP, and norms chew one row at a time, so doubling the context doubles their work. Attention’s score matrix and its weighted sum quadruple.

So the famous O(n²) is a crossover, not a flat tax. Per block the quadratic part costs about 4n²·d_model (the full n × n; a good kernel skips the masked half and pays half that), the linear part — projections plus MLP — about 24n·d_model². Equal at n ≈ 6·d_model: ~4,600 tokens for GPT-2 small, ~9,200 causal-only. Below that the MLP dominates; above it, attention is the bill. Nobody worried about this in 2019 with a 1024-token window; everybody does now.

Mixture of Experts

Mon, 01 Jan 0001 00:00:00 +0000

Replace the single MLP in a block with N of them — “experts” — plus a small router. The router scores the row, picks the top one or two, and only those run. Attention is untouched; MoE is a story about the MLP and nothing else.

The trade is parameters for FLOPs, and it’s a good one. Eight experts give a block eight times the MLP weights, but each row still runs through only one or two of them, so compute per token barely moves. Mixtral: 46.7B parameters total, 12.9B active per token — a top-2 number. Route to one expert and only ~7B would be active, at half the MLP FLOPs. You’re buying capacity — more places to put what the model knows — without buying much latency.

Superposition

Mon, 01 Jan 0001 00:00:00 +0000

A model keeps track of far more things than it has room for. Superposition is how it gets away with it: rather than giving every concept its own private slot, it overlaps them, cramming many more into the space than the space can strictly hold and accepting a little interference as the price. It works because on any given word almost none of those concepts are actually in play, so the collisions mostly don’t happen at the same time.

Fine-Tuning

Mon, 01 Jan 0001 00:00:00 +0000

Same architecture, same weights, more training — on your data, at a small learning rate. Nothing structural changes. Every virtue and every failure of fine-tuning follows from that.

What it’s good at is behaviour: format, tone, refusal style, the input-output shape of a task. Those are cheap to move, because the model already has the features and you’re only adjusting how it uses them.

What it’s bad at is facts. Teaching the model something it doesn’t know means shifting weights far enough to store new information, and the same gradient steps quietly degrade everything else — catastrophic forgetting. The rule of thumb that follows: if you want the model to know your documents, put them in the context; if you want it to behave a certain way, fine-tune.

RLHF

Mon, 01 Jan 0001 00:00:00 +0000

Cross-entropy trains a model to predict what a human would write. It has no way to say that one plausible continuation is better than another. RLHF is the machinery for training on that preference instead.

Three steps, classically. Supervised fine-tune on demonstrations. Then collect comparisons — a human sees two completions and picks one — and train a reward model: a copy of the language model with the unembedding swapped for a scalar head, trained so preferred completions score higher. Then optimize the policy against that reward with PPO, plus a KL penalty pinning it to the model it started from.

Speculative Decoding

Mon, 01 Jan 0001 00:00:00 +0000

Decoding is serial: each token depends on the last. Verifying isn’t. That asymmetry is the whole trick.

A small draft model generates k tokens (four, say) one at a time — but it’s small, so slow is cheap. The big target model then scores all k in a single forward pass, which it can do because that’s how training already works: one pass over a sequence yields a next-token distribution at every row at once. Compare the target’s distribution at each row against the draft’s, accept the longest prefix that survives a rejection-sampling test, resample the first rejected token from a corrected distribution, discard the rest.

Why Scale Worked

Mon, 01 Jan 0001 00:00:00 +0000

Modern language models are not good because someone discovered a smarter way to read a sentence. They are good because someone discovered a way to train on far more writing than anyone had trained on before — and it turned out that more writing was what the problem had been short of all along. The insight everyone quotes is attention. The insight that did the work is duller: this design can turn a bigger budget into more text read, and the design it replaced could not. This page argues the second one is load-bearing and the first is mostly a means to it.