Transformer

From text to the next token

A transformer is the machine inside almost every modern chatbot. It takes tokens in and gives a probability for every possible next token out. In between, the data flows through the same few parts again and again.

First every token becomes a vector, its embedding, and gets a mark of its position. Then the vectors pass through a stack of identical blocks. Each block lets the tokens exchange information with attention and then thinks about each token on its own with a feed-forward layer. At the end the vector of the last token is turned into scores for the whole vocabulary.

Try it. Click the parts of the diagram, or press play to send the tokens through.

Where is each word?

Attention by itself has no sense of order. To it, dog bites man and man bites dog are the same bag of tokens. So the model has to be told where each token stands.

The original transformer added a position code to every embedding, made of sine and cosine waves of different lengths. Each position gets its own pattern, shown as one row of the map. Most current open models, Llama among them, use rotary position embedding (RoPE) instead. It turns the query and key vectors by an angle that grows with the position, so the attention score between two tokens depends on how far apart they are, not on where they stand in the text.

Try it. Move the position slider and watch the arrow turn. Short waves on the left change fast, long waves on the right barely move.

Inside one block

A block of GPT-2 and most later models does four things to every token vector. It normalizes the vector so the numbers stay in a sane range. It runs attention and adds the result to the vector. It normalizes again. It runs the feed-forward layer and adds that result too.

The adding is the important trick. Every part only writes a small change on top of the vector, which is called the residual stream. Information from early layers can travel to the end untouched, and very deep stacks become trainable.

Try it. Step through the block. The weights here are random, so only the flow is meaningful, but every number is really computed.

The feed-forward layer

After attention, each token is processed on its own by a small network. It widens the vector, to four times its size in GPT-2 and three and a half times in Llama 3.1, bends it with an activation function and shrinks it back.

This part holds two thirds or more of the weights of a block. Research suggests that much of what a model knows about the world is stored here. Some neurons fire for a pattern in the input and push related tokens up in the output, like a giant key and value memory.

Try it. Compare the activations. GELU, used by GPT-2, is a smooth ReLU. Llama and many newer models use a gated variant called SwiGLU.

Where the billions come from

The size of a transformer follows from a few numbers. The width of the token vector, the number of blocks, the width of the feed-forward layer and the size of the vocabulary. In GPT-2 each block holds about 12 times the width squared in weights, 4 for attention and 8 for the feed-forward layer.

Current models change the recipe in two ways. Several query heads share one key and value head, which is called grouped-query attention and makes attention smaller. And many models, gpt-oss and Qwen3 among them, are a mixture of experts. Each block holds dozens or hundreds of feed-forward layers, and every token goes through only a few of them, so the model is big to store but cheap to run.

The formula lands within a few percent of the published sizes. Pick a preset and compare. The biggest open models of 2026, DeepSeek V4 Pro with 1.6 trillion parameters and Kimi K3 with 2.8 trillion, are mixtures of experts too. Closed models such as GPT-6 or Claude do not publish these numbers, but there is no sign that they are built differently.

Writing one token at a time

To write a reply the model runs the whole stack once for every new token. The new token needs the keys and values of all tokens before it in every layer. Recomputing them each time would be wasteful, so they are kept in memory, the KV cache.

The cache grows with every token in every layer. That is why long chats use a lot of memory on the server and why the first answer token, which has to process your whole prompt at once, takes longer than the following ones. Grouped-query attention helps here. Llama 3.1 8B keeps 8 key and value heads instead of 32, a quarter of the memory.

Paper

Attention Is All You Need, 2017

Block

norm, attention, add, norm, feed-forward, add (GPT-2 and later)

Parameters

(attention + feed-forward) × layers, plus the embeddings

Chat models

Decoder only, causal mask, KV cache