Neural networks

A neuron is a weighted sum

An artificial neuron is a tiny calculator. It takes a few numbers as input, multiplies each one by its own weight, adds them up together with one more number called the bias, and passes the result through a simple function.

Think of a neuron that decides whether to take an umbrella. One input says how cloudy it is, another what the forecast says. The weights say how much each input matters, and a negative weight means the input speaks against the decision.

Weights and biases are the parameters of a network. When people say a model has 70 billion parameters, they mean 70 billion of these numbers.

Try it. Move the sliders and follow the calculation. Make one weight negative and watch its line change color.

The bend that makes it smart

The function at the end of a neuron is called the activation. Without it, a neuron is a straight-line formula, and a stack of straight-line formulas is still one straight line, no matter how many layers you add.

A bend in the function changes that. The sigmoid squeezes any number into the range from 0 to 1, handy for a yes or no answer. Tanh does the same between −1 and 1. ReLU simply turns negative numbers into zero, and this cheap bend is what most large networks use, often in smoothed variants.

Try it. Switch between the functions. The dot is the sum from the previous step.

One neuron draws a straight line

Give a neuron two inputs, the position of a point on a plane, and its answer splits the plane in two. On one side the output is above 0.5, on the other side below. The border between them is always a straight line.

The weights turn the line and the bias moves it. Finding good weights by hand is fiddly even with three numbers. A real network has billions, which is why they are never set by hand.

Try it. Turn the sliders until the line separates the two colors. Wrong answers get a red ring. If you give up, ask for the best line.

Some problems need more than a line

Now the points form a ring, or four corners where the opposite ones belong together. No straight line can split these. Whatever the weights, one neuron gets a big part of the points wrong.

In 1969 the book Perceptrons showed this limit, with the XOR problem as the best known example, and interest in neural networks faded for years. The way out was already known in theory but hard to train at the time. Use more than one layer.

Layers combine lines into shapes

Put a few neurons side by side in a hidden layer and feed their answers into one more neuron. Each hidden neuron still draws one straight line, shown in the small squares. The output neuron combines the lines, and together they can enclose a ring or any other shape.

This network has four hidden neurons and was trained a moment ago in your browser. The next lesson shows how that training works.

Try it. Switch hidden neurons off one by one and watch the shape break.

From four neurons to billions

A language model is built from the same parts. Layers of weighted sums with bends between them, only very wide and very deep. Every connection between two neurons is one weight, so the count grows with the square of the layer width.

Our network has 17 parameters. The bars compare it with open models whose sizes are public. The largest of them passed a trillion parameters in 2026. Closed models such as GPT-6 or Claude do not publish their size.

Neuron

output = activation(w₁x₁ + w₂x₂ + … + b)

Parameters

All weights and biases of the network

Common bends

ReLU and its variants, sigmoid, tanh

First idea

The perceptron, 1958