Production RAG pipelines: where they fail and how to fix them

The pipeline, not the demo

Figure: Example trace in the Langfuse UI.

Most RAG demos look fine in a notebook. Production RAG fails in the seams between ingest, chunking, indexing, retrieval, generation, and evaluation. This guide is about those seams: what breaks after launch, how you notice, and what to change before users stop trusting the product.

A production RAG system is usually framed as separate pipelines that scale independently. A feature pipeline extracts, cleans, chunks, and embeds documents, then loads them into a vector database together with their metadata, usually on a schedule. A retrieval and inference pipeline embeds the user's query, fetches the most similar chunks, builds the prompt, and calls the LLM. Every query, enriched prompt, and answer then flows into a prompt monitoring system that can raise alerts.

A usable mental model is a straight line with feedback: documents → extract/normalize → chunk → embed/index → retrieve → generate → log & evaluate → re-ingest when truth changes. How often the vector store syncs with your source data has a direct effect on RAG accuracy. Skip the last loop and the system slowly lies.

Failure modes that show up in production

Retrieval drift: the corpus grew, embeddings aged, or the query mix shifted, so yesterday's "good enough" top-k is today's miss. Users feel this as confident answers that ignore a document they know exists. Fixes apply at three points. Before retrieval, route, rewrite, or expand the query, or use self-querying to pull out filters such as author or date. During retrieval, use hybrid search, filtered vector search, or fine-tuned or instruction-tuned embeddings. After retrieval, rerank a wider candidate set with a cross-encoder and keep only the top K.

Preprocessing skew: a subtle cause of drift is cleaning or embedding queries differently from documents. Both sides must go through the same preprocessing and the same embedding model, otherwise you get training-serving skew and silently worse matches.

Chunking regressions: a new splitter, PDF parser, or OCR path changes chunk boundaries. Related sentences get split across chunks, tables and lists collapse, and the model never sees the full rule. Useful mitigations are overlapping chunks, richer metadata on every chunk, and small-to-big retrieval, where you match on small precise chunks but pass the larger surrounding passage to the LLM.

Stale indexes: policies, SKUs, or SOP revisions land in SharePoint but not in the vector store. The system cites last quarter's truth with this quarter's confidence. Treat ingestion as a scheduled, monitored pipeline rather than a one-off import, and track how long it takes for a source change to become retrievable.

Hallucinated or weak citations: the model invents a source, points at the wrong passage, or answers from parametric knowledge while still rendering a citation UI. A solid generation pattern is to tell the model to answer only from the supplied context and to say it does not know when the context is insufficient. Output guardrails that check for hallucinations add a second layer of protection.

Cost and latency blowups: oversized context windows, multi-hop tool calls, or "always use the biggest model" routing turn a cheap assistant into a slow bill. Large or noisy context both costs more and distracts the LLM from the relevant facts. Reranking, prompt compression, and trimming to the top K chunks fix quality and spend together. Remember that guardrails and retries on invalid output also add latency.

How each failure shows up for users and operators

Users report: "It ignored the policy," "Citation doesn't match," "It was fast last month," or "Support got the same wrong answer twice." Operators see rising thumbs-down, repeat queries, escalation tickets, and token spend without a matching quality gain.

Map symptoms to stages. Wrong doc → retrieval or index freshness. Right doc, wrong snippet → chunking or ranking. Right snippet, wrong prose → prompt or model. A fancy citation that doesn't support the claim → grounding/citation check missing.

This mapping only works if you can see each step. Tracing matters for exactly this reason: if a query fails because of a wrong answer, an invalid context, or broken data processing, a full trace lets you point to the faulty step instead of guessing. Sudden swings in token counts at a specific step are a warning sign too.

What to log

Log enough to replay a bad answer without shipping PII you do not need. At minimum: request id, user/tenant id, query text, retrieved chunk ids and scores, model and prompt version, latency breakdown, token counts, citation ids, and user feedback.

Log the full trace, not just the final prompt and answer: the user input, the prompt template and its input variables, intermediate steps such as query rewriting and retrieval, the generated response, token counts, and latency. Prompt monitoring tools such as Opik, Langfuse, and Weights & Biases are built for this. Version prompt templates in Git, a database, or a prompt manager so every logged answer can be tied to the exact template that produced it.

For latency, track time to first token, time between tokens, tokens per second, time per output token, and total latency. Token counts are your cost signal. Keep raw retrieved text in a short retention window for debugging. Long term, keep ids, scores, and versions so you can rehydrate from the corpus. If you cannot answer "which chunks fed this reply?", you cannot improve the system.

Guardrails belong in the log too. Input guardrails catch private data heading to external APIs, jailbreaks, and harmful prompts. Output guardrails catch empty or malformed responses, toxicity, hallucinations, and sensitive-data leaks. Common tools include Galileo Protect and the OpenAI Moderation API. Record when a guardrail fires so you can measure its rate and the latency it adds.

How to evaluate without lying to yourself

Build a fixed set of real questions with expected sources and pass/fail rules. Score retrieval separately from generation. Run the set on every meaningful change to chunking, embeddings, or prompts.

Name the metrics for each side clearly. On the retrieval side, measure precision and recall of the retrieved context. On the generation side, measure whether the answer is faithful to that context and relevant to the question. Frameworks like Ragas implement these as context precision, context recall, faithfulness, and answer relevancy. Ragas can also generate synthetic test questions to grow your set and supports production monitoring. ARES is an alternative that combines synthetic data with trained classifiers to judge context relevance, faithfulness, and answer relevance.

Track regressions, not vanity. A 2% gain on a cherry-picked demo set means nothing if your production failure modes are unchanged. Pair offline evals with online signals: citation click-through, correction rate, "answer not found" handling, and thumbs-up/down feedback teams often collect, which can later feed preference datasets for fine-tuning.

Finally, treat degradation as something to detect, not something users discover. Good MLOps practice is to monitor for drift in the inputs and outputs, use proxy metrics when you have no labels, and wire alerts to retraining or re-ingestion through CI/CD and continuous-training pipelines.

Two production patterns

Applied Optoelectronics-style path: a Custom GPT / chat wrapper looked fine until RFP answers needed grounded, auditable passages from internal PDFs. Moving to a coded RAG pipeline with OCR, normalized chunks, and retrieval cut cost and raised answer usefulness compared with a brittle GPT-builder path. In practice, the win came from owning the ingestion pipeline, so cleaning, chunking, and metadata became controllable and testable.

Permit Signal-style path: an agentic RAG surface failed when retrieval and tool routing disagreed under real edge cases. Fixing production meant treating retrieval quality, guardrails, and eval harnesses as first-class, not bolting prompts onto a broken index. This matches standard advice to make query routing an explicit pre-retrieval step and to trace every intermediate decision.

Both stories share a lesson: when the product must be trusted, architecture beats prompt theater. If you are stuck between patching and rebuilding, a structured review of retrieval, chunking, evals, cost/latency, and model routing is the cheapest way to decide.

When to get a review

Book an AI App Review when the system is already live and unreliable: inconsistent answers, rising cost, missing evals, or citations nobody trusts. You leave with a written report and a prioritized fix list.

If the outcome is a clean rebuild on a fixed document set, the 100 Document RAG MVP is the usual next step: cited answers, measured accuracy, delivered in days, not another open-ended pilot.