Next-token prediction

A model only guesses the next token

A language model has one job. It reads the text so far and gives every token of its vocabulary a probability of coming next. Everything a chatbot writes, from a poem to program code, is this one guess repeated over and over.

The model on this page is a tiny one, trained in your browser on a short story of 600 words. It looks only at the last two words and counts what followed them in the story. A real model reads thousands of tokens of context through a neural network, but what comes out at the end is the same kind of list.

Try it. Click a word in the list to add it to the text and see the next guess. You are now the part of the program that picks tokens.

Scores become probabilities

A neural network does not output probabilities directly. Its last layer gives every token a raw score called a logit. Higher means more likely, and the scores can be any number, negative too.

A function called softmax turns the scores into probabilities. It raises the number e to the power of every score, so a higher score grows much faster, and then divides each result by their sum. All probabilities are positive and add up to exactly 1.

Our toy model takes the logarithm of each count as its score, so softmax gives back the plain shares of the counts. The table shows every step of the calculation for the text you built.

Temperature reshapes the guess

Before softmax, every score is divided by a number called the temperature. A temperature below 1 makes the differences bigger, so the favourite wins more often and the text gets safe and predictable. Above 1 the differences shrink, rare tokens get a real chance and the text turns creative, then strange.

As the temperature gets close to zero, the top token takes all the probability. Always taking the top token is called greedy decoding.

Try it. Move the slider to both ends. The dashed outlines show the original probabilities at temperature 1.

Top-k and top-p cut the tail

Even a good model gives a tiny probability to thousands of silly tokens. Together they add up, and one unlucky draw can derail a whole answer. Two simple filters cut that tail off before the draw.

Top-k keeps only the k most likely tokens. Top-p, also called nucleus sampling, keeps the smallest group of top tokens whose probabilities add up to p, so it keeps few tokens when the model is sure and many when it is not. The kept tokens are scaled up to add up to 1 again.

Try it. Set top-p to 0.5 and switch between a sure context like the robot and an open one like the.

One token at a time

Generation is a loop. The model guesses the distribution, the sampler draws one token with the temperature and filters you set, the token is added to the text, and the longer text goes back into the model. Nothing is planned ahead. A chatbot writes a long answer exactly like this, one token per step.

The color of each new word shows how likely it was. Bright tokens were safe picks, faded ones were long shots.

Try it. Generate a few sentences, then lower the temperature to 0.1 and try again. A greedy tiny model quickly starts going around in circles.

Why answers differ every time

Because the next token is drawn at random, the same prompt gives a different text on every run. With a temperature near zero the runs become almost the same. That is why chat apps give a fresh answer when you press regenerate.

It also explains made-up facts. The model picks tokens that are likely after the text so far, and nothing in the loop checks whether the result is true. A sentence can be smooth, confident and wrong.

Output

A probability for every token of the vocabulary

Softmax

p = e^{score / T} divided by the sum over all tokens

Typical settings

Temperature 0.2 to 1, top-p 0.9 to 1

Greedy

Temperature near zero, the top token every time