How a RAG pipeline moves from documents to an answer

Overview: two phases

A RAG pipeline works in two phases that run separately. The first is offline ingestion. It prepares your documents so they can be searched by meaning. The second is online retrieval and generation. It runs every time a user asks a question, finding the best passages and having a language model write an answer from them.

Keeping the phases separate is deliberate. The ingestion pipeline can run on a schedule on its own machine and keep the vector database up to date, while the retrieval step runs on demand for each request. The database stays fresh and answers stay fast.

Phase one: offline ingestion

Ingestion starts with data extraction, pulling documents from sources such as databases, APIs or web pages. A cleaning step then standardises the text, for example by removing unwanted characters or replacing URLs with placeholders.

Next comes chunking, which splits each cleaned document into smaller pieces. This is needed because embedding models accept limited input, and because smaller pieces that stay on one topic are easier to match to a question. Each chunk is then passed through an embedding model, which turns it into a dense vector that captures its meaning.

Finally, the loading step saves each embedding in a vector database together with metadata, such as the chunk's text, its source URL and the date it was published. The embedding is used to find the chunk, and the metadata is used to build the prompt and show the source.

Phase two: online retrieval

When a user asks a question, the retrieval step turns it into an embedding using the same model and the same cleaning rules that were used for the documents. This matters a great deal. If the question and the documents are not placed in the same vector space, their distances mean nothing, and the results will be poor.

The vector database then compares the question's vector with the stored vectors using a measure such as cosine distance and returns the top K most similar chunks. Databases use approximate nearest neighbour search for this, trading a little accuracy for a large gain in speed.

Phase two: generation

The generation step builds a prompt from a template, the user's question and the retrieved chunks. A typical template tells the model to answer using only the provided context and to say it does not know if the context is not enough. The augmented prompt is sent to the language model, and its answer goes back to the user. Track and version prompt templates so you always know which template produced a given answer.

Advanced steps that improve results

Production pipelines add optimisations around this basic flow. Before retrieval, query expansion rewrites the question into several variations, and self-querying extracts details such as an author name to use as a filter. During retrieval, filtered or hybrid search narrows or broadens the match. After retrieval, a re-ranker scores each candidate against the question and keeps only the best few. Each of these steps targets a common weakness of the simple version, and which ones help depends on your data.

Putting it together

In one line: ingest and index your data ahead of time, then for each question embed it, retrieve the closest chunks, optionally re-rank them, and have the model answer from them. The next guide breaks down each component in turn.