Tokenization

A model reads numbers, not letters

A language model never sees your text. Before anything else, a tokenizer cuts the text into pieces called tokens and swaps every piece for its number in a fixed vocabulary. The model reads only that list of numbers and writes its answer as numbers too.

Type anything into the box. The tokenizer on this page learned its vocabulary from the 5,000 most common English words, a small copy of how real tokenizers are made. Every colored box is one token, and the small number under it is the token ID the model would get.

Try it. Type a common word like the, then a rare one like photosynthesis. The common word is one token, the rare word falls apart into several.

Start from bytes

Every text on a computer is already a list of numbers. In UTF-8 an English letter takes one byte, a number from 0 to 255. So the simplest tokenizer has exactly 256 tokens, one per byte, and it can write down any text in any language.

Letters outside English take more bytes. A Cyrillic letter takes two, a Chinese character three, most emoji four. The boxes with hex codes like D0 are such bytes. They are not characters on their own.

The problem is length. With one token per byte a page of text becomes thousands of tokens, and a model pays for every one of them in time and memory. Byte pair encoding fixes that by teaching the tokenizer longer pieces.

Try it. Add a word with an accent or an emoji and count its boxes.

Count the neighbouring pairs

Training starts with a pile of text. The tokenizer splits it into words, keeping the space in front of a word as part of it (the ␣ mark), and writes every word as bytes.

Then it counts every pair of neighbouring tokens in the whole pile. A pair inside a word that appears 30 times counts 30 times. The table shows the most frequent pairs of a short training text, small enough to follow every rule, and the winner is at the top.

Nothing here knows grammar. The tokenizer finds frequent letter combinations only because they are frequent.

Merge the most frequent pair

The winning pair becomes a new token with the next free ID, 256, then 257 and so on. Everywhere the two pieces stand side by side they are glued into one. Then the counting starts again on the new pieces, and the next winner is merged.

Each merge is a rule, and the list of rules in order is the trained tokenizer. Common endings like ing and est and whole frequent words appear after a few dozen merges. Real tokenizers stop after tens of thousands of merges, between about 30,000 and 200,000.

Try it. Press the merge button a few times and watch the sample sentence shrink. Then drag the slider to the end.

Tokenize anything

The tokenizer of this step was trained the same way in advance, with 5,000 merges on the most common English words and how often they occur. To tokenize new text it replays its rules in the order it learned them. It starts from bytes and applies every merge that fits. Words it saw often in training end up as one token. New words are built from smaller known pieces, and in the worst case from single bytes.

That is why a model never meets an unknown word. It may not know what a word means, but it can always spell it.

Try it. Paste a sentence from any site. Then move the slider to the left to see how the same text looks with fewer merges.

Why token counts matter

Model prices and context limits are counted in tokens, not words. A text that splits into more tokens costs more, runs slower and fills the context window sooner.

Our tiny tokenizer learned only English, so other languages fall back to bytes and take many more tokens. Big tokenizers are trained on many languages and the gap is smaller, but it is still there. The same message in English is usually the cheapest, and code, numbers and rare scripts cost more.

Splitting into pieces also explains some famous model mistakes. A model that sees strawberry as two or three tokens does not see its letters, so counting the letter r is harder for it than it looks.

Method

Byte pair encoding (BPE)

Base tokens

256 bytes

Real vocabularies

about 32,000 to 200,000 tokens

Ours

5,256 tokens, word frequencies from wordfreq (CC BY-SA 4.0)

Used by

GPT, Llama, Mistral and most other modern language models