Retrieval-augmented generation: giving models the right context

Definition

Retrieval-augmented generation (RAG) is a technique that improves a language model's answers by fetching relevant information from an external source and adding it to the prompt before the model responds. The name describes the three actions. Retrieval means searching for relevant data. Augmented means adding that data as context to the prompt. Generation means using the enriched prompt with a language model to write the answer.

In practice, RAG lets a model answer questions about information it was never trained on, such as your company documents, last week's news or private records, without retraining the model.

The problems RAG solves

Two limits of language models explain why RAG is so widely used. The first is hallucination. A model asked about something outside its training data will often give a confident but wrong answer. With RAG, the model is told to answer only from the supplied context. The model acts as the reasoning engine and the retrieved text acts as the source of truth, so you can quickly check whether an answer is grounded in your data.

The second limit is old or private information. A model only knows what it was trained on, and training stops at a cutoff date. You also cannot train on data you do not own or have the right to use, and fine-tuning is too costly to repeat every day. RAG avoids the problem by injecting the needed data into the prompt at question time.

The three phases: ingest, retrieve, generate

A basic RAG system is usually described as three independent modules. The ingestion phase runs ahead of time. It pulls documents from sources, cleans them, splits them into chunks, converts each chunk into an embedding (a list of numbers that captures meaning) and stores the result in a vector database.

The retrieval phase runs when a user asks a question. The question is converted into an embedding in the same way, and the vector database returns the chunks whose embeddings are closest to it. The generation phase combines a prompt template, the user's question and the retrieved chunks into one prompt, sends it to the language model and returns the answer to the user.

A simple example

Imagine a support assistant for a software company. Overnight, the ingestion phase loads the latest help articles into a vector database. A customer asks how to reset a password. The retrieval phase finds the three most relevant help-article passages. The generation phase gives those passages and the question to the model, which writes a short, accurate answer based on them. When the help articles change, only the ingestion phase needs to run again.

What RAG does not do

RAG does not change the model's weights, so it does not teach the model new skills or a new writing style. It only supplies information. If the retrieval step returns the wrong passages, the model will answer from the wrong material, so most of the quality of a RAG system comes from how well it retrieves. For a step-by-step look at how the pieces fit, see our guide on how a RAG pipeline works.