The building blocks of a RAG architecture

The components at a glance

A RAG pipeline is made of a small set of parts, and each has one job. On the ingestion side they are data extraction and cleaning, chunking, an embedding model and a vector store. On the question-answering side they are the retriever, an optional re-ranker and the generator, which is the language model plus its prompt template. The sections below take them in order.

Data extraction and cleaning

Everything starts with getting documents out of their source, whether a data warehouse, a set of web pages or an API. A cleaning layer then standardises the text. How much cleaning you need depends on your sources and on the embedding model you use.

Chunking

Chunking splits documents into smaller pieces. It is required because embedding models have a maximum input size, and it helps because related text can be kept together so that only the essential content ends up in the prompt. Chunk size is a trade-off: very large chunks mix several topics and blur the meaning, while very small ones lose context.

Several techniques improve chunking. A sliding window overlaps neighbouring chunks so information near a boundary is not lost. Metadata such as dates, URLs and chapter markers allows filtering later. A small-to-big approach embeds a short passage for accurate matching but gives the model the wider surrounding text as context.

Embedding model

An embedding model converts text, and in some systems images or audio, into a dense vector so that similar meanings sit close together. Encoder models in the BERT family, used through libraries such as Sentence Transformers, are a common choice. The same model must embed both documents and queries. If your domain has special jargon, you can fine-tune the embedding model or use an instruction-guided embedding model as a lighter alternative.

Vector store

A vector database stores embeddings and finds the closest ones quickly. Unlike a standalone index such as FAISS, it also supports creating, updating and deleting records, metadata filtering, scaling, backups and security, which is why it suits production. Internally it uses approximate nearest neighbour algorithms such as HNSW to keep searches fast. Popular options include Qdrant, Milvus, Weaviate, Pinecone, Chroma and the pgvector extension for PostgreSQL.

Retriever

The retriever turns the user's question into an embedding and queries the vector store for the top K matches, using a distance measure such as cosine distance. It can also apply metadata filters, or combine vector search with keyword search in a hybrid search. Most of the advanced logic in a RAG system lives here, which is why retrieval gets so much attention in production designs.

Re-ranker

A re-ranker is a second, more precise pass over the retrieved chunks. It uses a cross-encoder model to score each chunk against the question together, which captures relationships a simple vector comparison can miss. It is too slow to run on the whole database, so the usual pattern is to retrieve a wider set of candidates cheaply and then re-rank them to keep only the best few.

Generator

The generator is the language model plus the prompt template. The template combines instructions, the user's question and the retrieved context, and the model writes the final answer. Because answers depend on the template, treat it like code: version it and test changes. Next, see our guide on how to build a RAG pipeline using these parts.