Break Down the RAG Pipeline Into 5 Product Decision Points

Break Down the RAG Pipeline Into 5 Product Decision Points

RAG (Retrieval-Augmented Generation) has 5 steps: Ingest → Chunk → Embed → Retrieve → Generate. For an engineer, these are 5 implementation steps. For a PM/BA, each step is a product decision point — a place where, if you don't weigh in, the engineer will pick a default, and that default may not match the experience you actually want.

1. The 5 RAG steps through a PM lens, not an engineering one

Look at each step for what it means to the product, not the implementation:

• Ingest (bringing data into the system) — Product decision: which source counts as "ground truth"? Internal docs, old FAQs, support tickets — which get ingested, which don't? Ingesting the wrong source means the AI answers based on information you never wanted it to use.
• Chunk (splitting long documents into smaller pieces) — Product decision: what unit should content be split on so an answer doesn't get "cut off" mid-way? (Lesson 17 goes deep on this.)
• Embed (turning a chunk into a numeric vector so the machine can compare similarity) — This is the most purely technical step; a PM doesn't need to intervene, just needs to understand: the system "understands" approximate meaning, not exact keyword matches.
• Retrieve (finding the chunks most relevant to a question) — Product decision: how many chunks are enough? What's the "relevant enough" threshold? (Lesson 18.)
• Generate (the AI writes an answer based on the retrieved chunks) — Product decision: what is the AI allowed to answer when no chunk is relevant enough? (Lessons 19 and 21.)

2. Example: an insurance Q&A chatbot and the decision point at each step

An app that answers policy questions for insurance sales agents (unrelated to any of Uy's own products, purely illustrative):

An agent asks: "Can a customer with type 2 diabetes buy the premium health insurance plan?"

• Ingest: is the exclusion-clause table ingested, or only the marketing brochure?
• Chunk: this clause sits inside an 800-word paragraph — if chunked at 200 words, the answer might only capture half the condition, missing the exception.
• Retrieve: the system pulls the top-3 most similar chunks — but "diabetes" and "chronic underlying condition" are two different phrasings; does retrieval catch that?
• Generate: if no chunk explicitly mentions type 2 diabetes, is the AI allowed to infer from "chronic conditions in general," or must it say "I'm not sure, please check with the underwriting team"?

3. From technical question to a line in the requirement doc

Every bracketed question in section 2 above is a line in the requirement doc, not a technical question you leave for the engineer to answer on their own. If you don't write it down first, the engineer will pick some default — and that default usually optimizes for build speed, not for the end user's experience.

4. Analogy: RAG as a junior research assistant

Think of RAG as a junior research assistant. You hand them a filing cabinet (ingest), they tab each page into small sections for easy lookup (chunk), learn to recognize which page "seems relevant" to a question (embed), pull out the few most relevant pages when you ask (retrieve), then write an answer based on those pages (generate). If the filing cabinet is missing documents, if they tab things sloppily, or if you never told them "say you don't know if you can't find it" — they will make things up.

Exercise 16.1: You're the PM for an internal chatbot that answers refund policy questions for a customer-support team at an e-commerce marketplace. The source data includes: (1) the official refund policy document (15 pages), (2) 200 old support tickets with sample answers, (3) an internal training slide deck that's a year out of date.

For each of the Ingest / Chunk / Retrieve / Generate steps, state one concrete product decision you need to make before the engineer starts coding (no technical detail needed — just "what decision" and "why it affects the support agent's experience").