Quantization

Every weight is a number in memory

A model is mostly a huge list of weights, and every weight takes space. Stored as a 32-bit float it takes 4 bytes, as a 16-bit float 2 bytes, as an 8-bit integer 1 byte and as a 4-bit integer half a byte.

Fewer bits mean fewer possible values. A float spends some bits on the exponent, which sets the range, and the rest on the mantissa, which sets the precision. BF16 keeps the range of FP32 with less precision, which is why it is popular for training. Integers have no exponent at all and need a separate scale.

Try it. Type a weight and see what each format actually stores.

Rounding to a few levels

Quantization stores the weights with fewer bits. The simplest method takes the largest weight, divides the range evenly into levels and rounds every weight to the nearest level. Only the small integers and one scale number are stored.

With 8 bits there are 255 levels and the rounding error is tiny. With 4 bits there are only 15, and with 2 bits just 3. The weights of a trained network sit mostly near zero, so most levels at the edges stay almost empty.

Try it. Lower the bits and watch the error grow.

One outlier spoils the scale

Large language models have a few weights and activations that are much bigger than the rest. With one scale for the whole matrix, that single giant stretches the levels so far that all the normal weights crowd into one or two of them.

The usual fix is to give every small group of weights, say 32 or 128 of them, its own scale. The outlier then ruins only its own group. Methods such as GPTQ and AWQ go further and use sample data to decide which weights need the most care.

Try it. Add the outlier, then make the groups smaller.

What it does to a network

Here is the small network from the training lesson, trained on the spiral a moment ago in your browser. Its weights are now rounded to fewer bits, and the map on the right shows what the rounded network answers.

At 8 bits you cannot see a difference. At 4 bits the border shifts a little. Below that the shape starts to fall apart. Big models usually take 4 bits better than small ones, because each single weight matters less.

Try it. Move the bits down step by step.

Quantization in practice

Quantization is how large models fit on ordinary hardware. A 70 billion parameter model needs about 140 GB in 16 bits but around 40 GB at 4 bits, small enough for a pair of gaming graphics cards or a large Mac.

Running locally, you meet it in file names. The GGUF files of llama.cpp come as Q8_0, Q5_K_M, Q4_K_M and so on, where the number is roughly the bits per weight. Lower numbers are smaller and faster, and at some point noticeably worse.

Formats

FP32, FP16, BF16, INT8, INT4

Basic method

Scale, round, store integers and the scale

Better methods

Per-group scales, GPTQ, AWQ

Rule of thumb

8 bits is nearly free, 4 bits is a good trade for big models