The RAG Pipeline: 5 Decisions That Control Answer Quality
RAG isn't one block - it's a pipeline with 5 decision points, and any one of them can break the answer regardless of which model sits at the end of it.
Most teams describe RAG the same way: "we added a vector database so the model can look things up." That description is technically true and almost useless - it skips every decision that actually determines whether the system answers correctly or confidently makes something up.
RAG is not one block you either have or don't. It's a pipeline with five separate decision points, each one capable of breaking the answer on its own, independent of which model sits at the end of it.
The RAG Pipeline High-Level Flow
Retrieval Is the Bottleneck, Not Generation
When a RAG system gives a wrong answer, the instinct is to blame the model - swap it for a newer one, tweak the prompt, add another instruction telling it to "be more careful." Most of the time, that's solving the wrong layer. The model can only work with what it was handed; if the right document was never retrieved, or was retrieved missing the context that made it relevant, no amount of prompting recovers it.
Academic work on RAG failure points backs this up: retrieval and representation failures - the document never surfaces, or surfaces without the context needed to use it - are the most common and best-documented category of RAG breakage, ahead of anything that happens once generation starts. Fix the pipeline stages before the model, not after.
The Same Pipeline, Four Different Ways to Fail
Each case below is a different real query hitting the same RAG pipeline - only one decision point was wrong each time.
What was Acme's Q3 revenue?
"...continued steady growth over the prior quarter, with margins improving slightly." (the sentence naming "Acme" and "Q3" was three chunks earlier, split away during preprocessing)
I couldn't find Acme's Q3 revenue in the available documents.
The chunk boundary fell between the identifying context (company, quarter) and the number itself. The retriever had no way to match the query to a chunk that no longer mentions what it's about.
Chunking Is a Design Decision, Not a Preprocessing Step
Every RAG pipeline splits documents into chunks before embedding them, and it's easy to treat that split as plumbing - something to configure once and forget. It isn't. Chroma's research into chunking strategies found that the choice of chunking method changes retrieval recall by several percentage points at the same chunk size, and the right strategy depends on the document: a structured contract chunks well with simple recursive splitting, a multi-topic research paper needs boundaries drawn around actual topic shifts, and a raw transcript needs neither.
A separate, easy-to-miss failure lives inside chunking rather than around it: a chunk can be the right length and still lose the context that makes it findable. A paragraph about a company's revenue means nothing to a retriever once the sentence naming the company and the quarter has been split into a different chunk. Anthropic's engineering team measured this directly - prepending a short piece of context to each chunk before embedding cut retrieval errors by 49% in their tests, without changing which model answered the question.
Recall Has to Come Before Precision
Once chunks are indexed, retrieval quality comes down to two competing numbers: recall (did the relevant document come back at all) and precision (how much of what came back is actually relevant). Turning up how many results you retrieve raises recall and lowers precision - more of the true answer shows up, but so does more noise around it.
The ordering here matters more than the exact tradeoff point. If the right document was never in the retrieved set, no reranking model, no better prompt, and no bigger context window fixes that - the answer was never possible. Precision problems are fixable after the fact by filtering; recall problems are not fixable after the fact at all, because the information genuinely isn't there anymore by the time the model sees it.
Three Measured Numbers, Three Independent Experiments
They don't sum together (different experiments, different baselines) - but they all point the same direction.
Sensible default for most use cases (Chroma Technical Report)
Marginal gain, at the cost of embedding every sentence (Chroma Technical Report)
Adds context before embedding, independent of chunking strategy (Anthropic Engineering)
All three numbers come from separate benchmarks and can't be added into one total - but each measures the impact of a data-preparation decision, not a model swap. That's why the five decisions in this piece deserve the same care as picking a model.
Grounding Is Not the Same as Being Correct
Retrieval and grounding fail in different ways, and conflating them hides the second one. A grounded answer is one that's actually supported by the context it was given - not one that happens to be true. A model can retrieve the exact right passage and still overstate what that passage says, turning "not officially tested, but some users report it works" into a flat "yes, it's compatible."
That distinction is why "answer only using the retrieved context" has to be an explicit instruction and an explicit check, not an assumption. It's checkable at the level of individual claims: break a generated answer into its component statements and verify each one is actually supported by the specific chunk cited for it, rather than judging the answer as a whole - a plausible-sounding paragraph can contain one unsupported claim sitting next to three well-grounded ones.
Freshness Is a Product SLA, Not a Technical Afterthought
An index that answered correctly last month can be quietly wrong today if the underlying documents changed and nothing told the index to catch up. The mistake is treating "how often do we re-index" as one global setting. It isn't - a pricing page and a background reference document carry completely different costs when they're stale, and a single blanket refresh schedule either wastes compute re-indexing content that never changes or leaves fast-moving content wrong for days.
The fix is to size the freshness requirement to the actual cost of being wrong: near-real-time for anything tied to price or policy, hours for content that changes often but isn't safety-critical, and a much longer window for reference material that's stable by nature. That's a product decision about acceptable risk, not a database configuration setting.
Five Decisions, One Pipeline
Each of the four failure modes above traces back to one of five decisions that shape a RAG pipeline end to end - and each one deserves its own deeper look:
Chunking strategy. How to match a chunking method to a document's actual structure, and why chunk size alone isn't the variable that matters most.
Recall vs. precision. How to define what "good retrieval" means for a specific feature, and why recall has to be fixed before precision is worth tuning.
Grounding contract. How to write and enforce a rule that keeps generated answers inside the bounds of what was actually retrieved.
Indexing cadence. How to set a freshness SLA per data type instead of one global re-indexing schedule.
Failure mode taxonomy. How to classify where a RAG system is actually breaking - retrieval, representation, or generation - before trying to fix it.
The Shape of a Working RAG Pipeline
Strip away the specific techniques, and a working RAG pipeline gets three things right: it retrieves the right document before worrying about how prettily the answer is generated, it keeps every generated claim traceable to something it was actually given, and it treats staleness as a risk to size per data type rather than a single knob to set once. Get those three right, and the model you put at the end of the pipeline becomes one of the least important decisions in the whole system - which is exactly the point.