Chunking Strategy: The Silent Cut That Breaks Retrieval
A bad chunk boundary doesn't throw an error - it just means the correct chunk never gets retrieved, and you'll look everywhere except where the problem actually started.
Part of: The RAG Pipeline: 5 Decisions That Control Answer Quality
The default way to chunk a document is to pick a token count - say 400 - and cut every N tokens until the document runs out. It's cheap, it's deterministic, and it's the first thing every RAG tutorial shows. It's also the reason a correct answer sits three tokens away from the chunk that got retrieved, split cleanly in half by a boundary that has no idea what a sentence, a table row, or a section header is.
Chunking isn't a preprocessing step you configure once and forget - it's a design decision that determines what the retriever can ever find, no matter how good the embedding model or the reranker downstream is.
The Chunking Decision Flow
Why the Default Chunking Choice Quietly Breaks Retrieval
Fixed-size, token-count chunking treats a document as an undifferentiated stream of tokens. It has no concept of where one idea ends and another begins, so it cuts wherever the counter happens to hit zero - mid-sentence, mid-table-row, between a clause and the number it refers to. The failure this produces is silent: nothing errors out, the pipeline runs end to end, and the retriever returns a chunk that looks plausible. It's only when you trace a wrong answer back to its source chunk that the cut becomes visible - and by then it looks like a retrieval or generation bug, not a chunking one.
Recursive vs. Semantic Chunking: What the Numbers Actually Show
Two alternatives fix the boundary problem in different ways. Recursive chunking still targets a token budget, but it tries splitting at natural breakpoints first - paragraph, then sentence, then word - only falling back to a hard cut when nothing else fits. Semantic chunking goes further: it measures embedding similarity between consecutive sentences and cuts where the topic actually shifts, regardless of token count. Chroma's 2024 evaluation of chunking strategies measured this directly: recursive chunking at 400 tokens landed around 85-90% recall, semantic chunking pushed that to roughly 91-92% - at the cost of embedding every sentence individually before you can even form a chunk. The gap is real but not enormous, which is itself the finding: respecting any natural boundary buys most of the improvement over a hard token cut, and semantic chunking's extra gain has to be weighed against its extra embedding cost.
When Chunk Size Isn't the Real Problem, Structure Is
Token-boundary fixes don't help when the document itself isn't prose. A pricing table, a spec sheet, a legal document with numbered clauses - these have structure that a sentence-aware splitter still doesn't see. A table row split across two chunks separates a label from its value just as thoroughly as a fixed cut does; a clause number separated from the clause it introduces is unreadable in isolation. For structured content, the chunking boundary needs to align with the document's actual structure - table boundaries, section boundaries, clause boundaries - not with a token count or even a sentence boundary.
Chunk Boundary Failure Inspector
How long is the termination notice period?
"...Party B may terminate the agreement if Party A defaults on payment for more [CHUNK ENDS HERE, TOKEN 400] than 30 days from the due date..."
The document does not specify a termination notice period.
Fixed-size chunking cuts exactly at token 400 regardless of sentence boundaries — the clause is split mid-sentence, and the second half (containing '30 days') lands in the next chunk, which was never retrieved.
→ Switch to recursive chunking (respects sentence/paragraph boundaries)
Context Loss Is a Separate Failure From Bad Boundaries
A chunk can respect every sentence and table boundary perfectly and still fail, because the sentence itself depends on context that lives outside the chunk. "Revenue grew 12% quarter-over-quarter" is a complete, well-formed sentence - and useless on its own, because it doesn't say which company or which quarter. Anthropic's Contextual Retrieval work names this as a distinct failure mode from boundary-cutting: the chunk is internally coherent but was severed from the document-level context (company name, report period, section title) that gives it meaning. Their fix is mechanical rather than architectural - prepend 50-100 tokens of that context to each chunk before embedding it - and it cut retrieval errors by 49% in their evaluation. This is a different lever from choosing recursive vs. semantic splitting; it's worth applying regardless of which strategy you pick.
Matching Strategy to Document Type and Question Type
There's no universally correct chunk size or strategy, because the right answer depends on two things that vary per document: how the document is structured, and what kind of question it needs to answer. Dense narrative prose - a report, an article - tolerates recursive chunking well, since paragraph and sentence boundaries mostly line up with idea boundaries. Structured documents - contracts, pricing pages, API references - need structure-aware splitting that respects tables, clauses, and headers over token count. Questions that need a single fact answered well with a tightly-scoped chunk; questions that need synthesis across a broader passage need chunks large enough to hold that passage without being retrieved as disconnected fragments. Treat chunk size and strategy as a decision to revisit per document type, not a global default set once at pipeline setup.
Recall by Chunking Strategy
Chunking strategy → % of queries where the correct chunk lands in the top-k results
Ignores sentence/paragraph/table boundaries — prone to splitting related information mid-thought.
Still targets a token size, but prefers natural break points at paragraph/sentence boundaries.
Splits at meaning boundaries (embedding similarity between sentences) — higher recall, at the cost of embedding each sentence individually.
Prepending 50-100 tokens of context (document/section name) before embedding each chunk — independent of the cutting strategy chosen above — cuts49% of retrieval errors
These three numbers come from two separate experiments (Chroma measures recall by cutting strategy; Anthropic measures the error reduction from adding context) — they don't sum into one figure. Read them as two independent levers: pick a better cutting strategy, and add context to each chunk — both raise recall, but through different mechanisms.
Signals That Your Chunking Strategy Needs Revisiting
A chunking strategy that worked at launch can quietly stop working as the document set changes - a new document type gets ingested, an existing document gets restructured, or a new class of question starts showing up in real traffic. The signal to watch isn't a vague sense that answers feel off; it's a recall metric (see the recall/precision tradeoff spoke in this series) trending down on a specific document type or question category, or a recurring pattern in error analysis where the correct chunk was in the index but never made it into the retrieved set. Either signal points at a boundary problem before it points at anything downstream.
Common Pitfalls in Chunking Strategy Decisions
A few mistakes recur across RAG pipelines: picking a single global chunk size and strategy for a document set that actually contains several structurally different document types; treating semantic chunking as a strictly-better upgrade without weighing its added embedding cost against the real recall gain it buys; fixing boundary problems while leaving context loss unaddressed, since they're separate failure modes with separate fixes; and never revisiting the chunking decision once it's shipped, even as the underlying document set changes shape. Chunking is a design decision made early in the pipeline, but its effects surface late - in every wrong answer whose real cause was a cut made months earlier, three steps upstream.