Text and images

Pictures with captions

How does a model learn that a photo of a dog and the words a dog mean the same thing? In 2021 OpenAI trained CLIP on 400 million pictures from the internet together with the text that stood next to them, such as captions and alt texts. Nobody labelled anything by hand.

Our toy world is much smaller. Twelve pictures of colored shapes by day and by night, each with a short caption. The pictures are drawn by your browser.

Two encoders, one space

CLIP has two encoders. One turns a picture into a vector, the other turns a text into a vector of the same size. Training pushes the vector of a picture and the vector of its own caption to point the same way, so pictures and words end up in one shared space.

Our encoders are hand-written but honest. The picture encoder really reads the pixels. It measures which colors are in the shape, how the shape fills its box and how bright the background is. The text encoder looks for words it knows. Real CLIP learns both encoders from data and uses 512 or more numbers.

Try it. Pick a picture and type a caption. Then try a wrong one.

Training: match the right pairs

Training takes a batch of pictures and their captions and computes the similarity of every picture with every caption. The right pairs lie on the diagonal of this table. The loss rewards a high diagonal and punishes high values everywhere else, so every picture must pick its own caption out of the whole batch. This is called contrastive learning.

Before training the encoders are random and the table is noise. After training the diagonal lights up.

Try it. Switch between random and trained encoders and compare the loss.

Classify without training

A trained CLIP can sort pictures into classes it was never trained on. Write each class as a short text, such as a photo of a star, compute the vectors, and pick the text that fits the picture best. This is called zero-shot classification.

Change the list and you have a new classifier, without a single new training example.

Try it. Edit the labels, for example turn them into colors.

Search pictures with words

The same trick runs backwards. Store the vector of every picture once, turn the search text into a vector and return the closest pictures. Photo search can find pictures of a beach or a birthday cake this way without any tags.

CLIP-style encoders also sit inside other models. Stable Diffusion 1 read its prompt with the CLIP text encoder alone. Newer image generators such as Stable Diffusion 3 and FLUX.1 add the large T5 text encoder to CLIP, which understands long prompts much better. Many chatbots that understand images use a CLIP-like picture encoder.

Model

CLIP, OpenAI, 2021

Training data

400 million picture and text pairs

Method

Contrastive learning, matching pairs in a batch

Used in

Image search, zero-shot classification, image generators, vision chatbots