Convolution

A picture is a grid of numbers

To a computer a black and white picture is a table of brightness values, one number per pixel, here from 0 for black to 9 for white. A color picture has three such tables, for red, green and blue.

An image model starts from nothing but these numbers. It has to find edges, shapes and finally objects in them by itself.

Try it. Draw on the grid with the mouse or a finger and watch the numbers change.

A filter slides over the picture

A convolution uses a small table of weights called a filter or kernel, often 3 by 3. The filter is laid over a patch of the picture, each weight is multiplied with the pixel under it, and the nine products are added up. The sum becomes one pixel of a new picture called a feature map. Then the filter moves one pixel on and does it again.

This filter has negative weights on the left and positive ones on the right. It gives a big number where the picture gets brighter from left to right, a vertical edge, and nearly zero on flat areas.

Try it. Step the filter along, or play the whole pass. Your drawing from the last step is the input.

Different filters find different things

Change the nine weights and the same sliding finds something else. An even average blurs. A strong center with a negative ring sharpens. Other patterns find horizontal edges, vertical edges or outlines in every direction.

Photo editors used hand-made filters like these for decades. A convolutional network does not get them from a person. It learns the weights of hundreds of filters in training, and the first layer usually ends up with edge and color detectors much like these.

Try it. Pick a preset, then type your own weights.

Keep the strong signals, shrink the map

After a convolution, an activation such as ReLU sets the negative values to zero, so only the found features remain. Then pooling shrinks the map. Max pooling keeps only the largest value of every 2 by 2 square, so the map gets four times smaller while the strong signals survive.

Shrinking also makes the network a bit forgiving. An edge that moves by one pixel still lands in the same pooled square.

Deeper layers see bigger areas

A network stacks many such layers. Each pixel of a deep feature map depends on a bigger patch of the input, its receptive field. The first layers see a few pixels and react to edges. Later layers combine edges into corners and textures, then into parts such as eyes or wheels, and the last layers into whole objects.

AlexNet won the ImageNet contest by a wide margin in 2012 with this design and started the deep learning boom. Convolutional networks still run in many phones and cameras. Newer image models often use transformers instead, which cut the picture into patches and treat them like tokens.

Filter

A small table of learned weights, often 3 × 3

Layer

Convolution, activation, sometimes pooling

Milestones

LeNet-5 1998, AlexNet 2012, ResNet 2015

Successor

Vision transformers, from 2020