Mixture of experts

Many specialists, few at work

In a normal transformer every token passes through every weight of the feed-forward layer. A mixture of experts (MoE) splits that layer into many smaller networks called experts and adds a small router in front of them.

For each token the router picks only a few experts, often two out of eight or eight out of hundreds. The other experts do nothing for that token. The model can hold a huge amount of knowledge while each token pays for only a small part of it.

Try it. Press play and watch each token light up its two experts.

The router scores every expert

The router is a single small layer. It multiplies the token vector by its weights and gets one score per expert. Softmax turns the scores into shares, the top k experts are kept, and their shares are scaled up to add up to 1 again.

Each chosen expert processes the token, and the outputs are mixed in those shares. Our toy router uses six readable features of a token. A real one reads the whole hidden vector of thousands of numbers.

Try it. Pick a token and change k.

Each token takes its own route

Routing happens per token and per layer, so one sentence spreads over many experts. In our toy the experts were given clear jobs such as numbers, punctuation or code, so the routes are easy to read.

Real experts are messier. The Mixtral authors found no clear topic experts, such as one for biology or one for law. The patterns they saw followed the form of the text, such as code syntax or indentation, more than its subject.

Try it. Type your own sentence.

Keeping all experts busy

Left alone, a router tends to fall in love with a few experts. They get most of the tokens and train more, which makes the router like them even more, while the rest sit idle. Idle experts waste memory, and overloaded ones slow everything down.

Training fixes this with load balancing. Older models added an extra loss that punishes uneven use. DeepSeek-V3 instead adds a small bias to each expert's score, raised for idle experts and lowered for busy ones. The simulation does exactly that on 2,000 tokens.

Try it. Run a few balancing rounds and watch the bars even out.

Big to store, cheap to run

An MoE model has two sizes. The total size decides how much memory it needs. The active size, the weights one token actually uses, decides how fast and cheap each token is.

Mixtral 8x7B stores 46.7 billion parameters but uses about 12.9 billion per token. Qwen3 235B-A22B carries both numbers in its name. OpenAI's open gpt-oss-120b stores 117 billion and uses 5.1 billion. The largest open models of 2026 follow the same plan. DeepSeek V4 Pro stores 1.6 trillion and uses 49 billion, Kimi K3 stores 2.8 trillion and uses 104 billion.

Replaces

The feed-forward layer of a block

Router

One small layer, softmax, top-k

Mixtral 8x7B

8 experts, 2 per token, 46.7B total, 12.9B active

DeepSeek-V3

256 routed experts, 8 per token, 671B total, 37B active

Qwen3 235B-A22B

128 experts, 8 per token, 235B total, 22B active