Embeddings

A word becomes a list of numbers

An embedding is a list of numbers that stands for a word, a sentence or a whole document. The list is called a vector, and every number in it is one coordinate. Texts with similar meaning get similar lists.

Our toy model gives every word 8 numbers, and we named each coordinate so you can read it. One says how royal the word is, another whether it is male or female, another whether it is food. Pick a few words and compare their bars.

Real models learn their coordinates on their own from billions of sentences, and the coordinates have no names. A modern embedding model uses from a few hundred to a few thousand numbers per text, 768, 1536 and 3072 are common sizes.

Try it. Compare king and queen. Only one bar is really different.

Similar words land close together

Eight numbers are eight directions, too many to draw. To get a picture we lay the words out on a flat map so that their distances on the map stay as close as possible to their real distances. This is called multidimensional scaling, and your browser computes it each time the page loads.

Groups appear by themselves. Animals sit together, food sits together and vehicles keep their own corner. People split into men and women, with the royals right beside them. Nobody drew these groups. They come from the numbers.

Try it. Click a word. Lines lead to its three nearest neighbours, measured in all eight dimensions, not on the flat map.

Closeness is an angle

Think of every vector as an arrow from the origin. Two texts are similar when their arrows point the same way. The usual measure is cosine similarity, the cosine of the angle between the arrows.

A cosine of 1 means the same direction, 0 means unrelated, and negative values point away from each other. Length does not count, so a short and a long text about the same topic still match.

Try it. Compare kitten with puppy and then with bus. The picture is drawn in the flat slice through both arrows, so the angle you see is the real one.

Arithmetic with meaning

Directions carry meaning, so you can add and subtract them. Take king, remove man and add woman. The royal part stays, the male part is swapped for the female one, and the nearest word to the result is queen.

The picture shows the flat slice through the three words, where the sum is exact. The three input words are left out of the answer, as in the original word2vec experiments. Real embeddings are noisier than our toy, so this trick works there only for strong, frequent relations.

Try it. Try the presets, then build your own. What is puppy minus dog plus cat?

Search by meaning

This is what embeddings are used for every day. Every document gets a vector once. A search query gets a vector too, and the search returns the documents whose arrows point the most the same way. No keyword has to match.

Here a document vector is simply the average of the vectors of its known words, which is how early systems did it. Modern models read the whole sentence at once, but the search step is the same. RAG systems, recommendations and duplicate finders all work like this.

Try it. Search for kitten. The top document is about a puppy and a cat, and the word kitten is not in it.

Measure

Cosine similarity

Real sizes

A few hundred to a few thousand numbers per text

Stored in

Vector databases and search indexes

Used for

Semantic search, RAG, recommendations, clustering, duplicate detection