Fine-tuning and LoRA

Three ways to change a model

There are three common ways to make a model do what you need. Write better instructions in the prompt. Give it the facts at question time with RAG. Or train it a little more on your own examples, which is called fine-tuning.

They solve different problems. Fresh or private facts belong in RAG, because a fine-tuned model still makes up facts it saw only a few times. Fine-tuning is for habits, such as a fixed output format, a house style, a narrow skill or a shorter prompt for the same result.

Try it. Pick a goal and see which tool fits.

Full fine-tuning moves every weight

The plain way is to continue training the whole model on new examples. Every weight gets a gradient and changes a little. It works, but the bill is large.

During training the computer has to hold much more than the weights. Common mixed-precision training with the Adam optimizer keeps about 16 bytes per parameter, for the weights, their gradients, the optimizer state and a full-precision copy. And every fine-tuned version is a complete copy of the model.

Try it. Move the slider to see how much memory training needs for models of different sizes.

LoRA trains a thin patch

LoRA, short for low-rank adaptation, freezes the original weights. Next to a big weight matrix it trains two thin ones, B and A. Their product B · A has the same shape as the big matrix and is added to it.

The inner size r, the rank, is tiny, often 8 or 16. So the two thin matrices hold a small fraction of the numbers of the big one. The original LoRA paper from 2021 cut the trained parameters of GPT-3 by about 10,000 times.

Try it. Change the rank and compare the counts.

Why a thin patch is enough

A product of thin matrices can only describe changes with a simple structure, a few directions at a time. It turns out that the changes fine-tuning needs are mostly like that. The LoRA authors showed that the weight updates of fine-tuning have a low intrinsic rank.

The left map is a change made of three directions plus some noise. The right map is the best possible copy with rank r, found by singular value decomposition, computed in your browser. Three directions are enough to catch almost all of it.

Try it. Move the rank from 1 to 3 and watch the error drop.

Training a LoRA

Here a frozen layer of 8 by 8 weights has to learn a new task. The change it needs has rank 2. Only A and B are trained, with the gradient descent from the training lesson.

B starts at zero, as in the LoRA paper, so before training the patched model is exactly the original. With rank 1 the patch can learn only one of the two directions and the loss gets stuck. With rank 2 or more it reaches zero.

Try it. Train with rank 1, then with rank 2.

One model, many adapters

A LoRA adapter is a small file. One copy of the base model can stay in memory while adapters for different customers or tasks are swapped in, or even served side by side.

QLoRA, from 2023, goes further and keeps the frozen base model in 4-bit numbers during training. Its authors fine-tuned a 65 billion parameter model on a single 48 GB graphics card.

Full fine-tuning

All weights, about 16 bytes per parameter during training

LoRA

W + B · A, original weights frozen

Typical rank

4 to 64

Good for

Format, style, narrow skills, not fresh facts