RAG

Search first, then answer

A language model knows only what was in its training data, and that data ends on some date. It has never seen your price list, your manual or yesterday's news. Ask about them and it either says it does not know or, worse, makes up a smooth and wrong answer.

Retrieval augmented generation, RAG for short, fixes this without retraining. Before the model answers, a search finds the passages of your documents that fit the question and pastes them into the prompt. The model then answers from text it can actually read.

Our example is the owner handbook of a made-up e-bike. Pick a question and compare the two prompts the model would receive.

Cut the documents into chunks

A whole manual is too long to paste into every prompt, and search works better on short pieces. So the documents are cut into chunks once, in advance. A common choice is a few hundred tokens per chunk, cut at sentence or paragraph borders.

Size is a trade-off. Small chunks match a question precisely but may cut an answer in half. Big chunks keep the context but blur the match. Overlap repeats a sentence or two at each border, so an answer that falls on a cut still lands whole in one chunk.

Try it. Change the size and the overlap and watch how the handbook is cut.

Turn every chunk into a vector

Every chunk gets an embedding, a vector that describes its meaning, as in the embeddings lesson. The vectors are stored in an index, often a vector database. At question time the question gets a vector from the same model.

Our embedding is a toy you can read. It counts about two dozen topics, each defined by a hand-made list of words, so that send back and refund both count as returns. Real embedding models learn such links from data and use hundreds or thousands of numbers.

Try it. Type a question and see which topic columns light up.

Find the closest chunks

The search compares the question vector with every chunk vector by cosine similarity and returns the top few. This finds chunks that talk about the same thing in other words.

Plain keyword search still matters. It is better at exact names, numbers and codes. The classic ranking formula is BM25, which rewards rare words that appear in both the question and the chunk. Many systems run both searches and merge the results, which is called hybrid search.

Try it. Ask whether you can plug it in when it is freezing outside. None of these words appears in the handbook, so the keyword search scores zero everywhere, while the meaning search finds the charging rules.

Build the prompt

The best chunks are pasted into the prompt together with the question and a few instructions. The instructions matter. Answer only from the sources, cite them, and say so when the answer is not there.

This is the text a model would receive. RAG adds no magic to the model itself. It changes only what the model gets to read.

Try it. Copy the prompt and paste it into any chatbot.

Where RAG goes wrong

Many RAG failures happen in the search, not in the model. The right passage was cut in two, or it never made it into the top results, or the documents simply do not hold the answer and the model is handed loosely related text instead.

The chart runs the search for every chunk size and shows whether the passage with the answer reached the top three. Good systems test their search like this on a set of real questions, add a second model that re-ranks the results, and tell the model plainly to admit when the sources are silent.

Steps

Chunk, embed, store, retrieve, prompt, answer

Search

Vectors by cosine similarity, keywords by BM25, or both

Good for

Manuals, support bases, internal documents, fresh data

Not for

Teaching a model a new style or skill, which needs fine-tuning