Why Chunking Decides What Retrieval Can Ever Find
A retriever cannot return a passage that does not exist as a chunk. That is the whole argument. Everything after chunking — the embedding model, the reranker, the prompt, the model itself — chooses from a menu that chunking wrote. And if the one sentence that answers a question sits in a chunk whose other 700 tokens are about something else, the vector for that chunk is an average of both topics. Averages sit near everything and close to nothing.
So here is the honest answer before the taxonomy: for most business corpora, the winning strategy is to follow the structure the document already has, and to stop tuning chunk size. The elaborate strategies teams reach for are usually compensating for a parser that destroyed that structure before the chunker ever saw it. Chunking arguments are, more often than not, parsing problems in disguise.
Chunking also deserves more care than a normal parameter because it is sticky. Change the boundaries and every vector in your index is invalid: you re-embed the whole corpus and re-run your evaluation. On a few thousand documents that is an afternoon. On a machine builder’s manual archive it is a batch window and a bill.
Three Chunking Strategies, Ranked
Fixed-size, semantic, and structure-aware splitting dominate the discussion. They are not equals, and the middle one attracts far more attention than it earns.
Fixed-size splitting: the baseline you have to beat
Cut every N tokens, add a little overlap, move on. LangChain’s RecursiveCharacterTextSplitter is the common form: it breaks on paragraphs, then lines, then sentences, then characters. It is deterministic, fast, free, and trivially debuggable — “why is this chunk here?” always has an answer. For corpora with no structure to respect, it is not a compromise but the correct answer: call transcripts, chat logs, scraped forum threads, OCR from handwritten shop-floor notes. Its failure mode is at least loud: it severs a numbered clause from its number and splits a procedure between step 4 and step 5.
Semantic chunking: the one I would not start with
Embed each sentence, measure the distance to the next, cut where the distance spikes. In principle this finds topic boundaries the formatting never marked. In practice, on the corpora I actually see, it mostly rediscovers paragraph breaks — which the document handed you for free, as a plain-text marker. And it charges for the rediscovery: an embedding call per sentence at ingest, across an archive of 900-page manuals, is a real bill and a real batch window.
The deeper problem is that it is non-deterministic in the way that matters. Your boundaries become a function of the embedding model, so upgrading that model silently reshapes the index. And when you sit down to debug a bad answer, “why does this chunk start here?” has no readable answer. I would not rule it out for the case it suits — long unstructured prose where topics drift with no formatting cue, such as interview notes — but I would not make it a default, and never before there is an evaluation set that can prove it earned its cost.
Structure-aware splitting: use the skeleton the author wrote
Documents that matter to a business are almost never a flat wall of text. A supply agreement has numbered paragraphs. A maintenance manual has chapters, sections, and numbered procedures. A regional insurer’s policy conditions have clauses a lawyer spent real time arranging. Someone already did the semantic segmentation — by hand, with headings — and encoded the result in the document. Split on those boundaries and you inherit that work: a Markdown heading splitter, an HTML heading walk, the outline in a DOCX.
Which is why the real work sits upstream. Structure-aware chunking is only possible if the parser preserved the structure, so convert to a structured intermediate — Markdown or HTML — with a tool built for it, such as Docling, Unstructured, or Azure Document Intelligence, rather than dumping a PDF to plain text and losing the outline in step one. A team that flattens its PDFs then needs semantic chunking to reconstruct what it just discarded. This is the same unglamorous groundwork as preparing your data for AI, and it pays out the same way.
The useful framing: structure-aware chunking is fixed-size chunking with the boundaries chosen by the author instead of by a modulo operation. The two are not rivals. You split on structure, and where one section still runs past your ceiling, you split that section by length.
Stop Choosing a Chunk Size
The size debate — 300 tokens or 1,000? — assumes one number must serve two jobs that pull in opposite directions. Retrieval wants small: an embedding is a single fixed-length vector, so the more distinct ideas you compress into one chunk, the blurrier that vector gets. Generation wants big: the model needs the definition two paragraphs up and the exception in the sentence below, or it answers confidently from half the picture.
You do not have to pick, because the chunk you retrieve does not have to be the chunk you send. Index the small unit, keep a pointer to its parent section, and on a hit, expand the match and hand the model the parent. LlamaIndex ships this as auto-merging retrieval, LangChain as ParentDocumentRetriever, and it is about forty lines if you would rather own the code than the dependency.
The cost is real and worth naming: five small hits can expand into three parent sections and a lot of tokens, so you deduplicate parents and cap the context budget. But it dissolves the parameter everyone argues about, and it is the single change that most reliably improves a mediocre RAG system I am handed.
Overlap, Tables, and the Line That Rescues a Chunk
Overlap gets treated as a default — 10 to 20 percent, why not. It is a patch for boundaries you do not trust. It costs index size, and it costs you top-k slots: the same sentence ranked first and second means one idea eating two of the five seats in your context. When boundaries follow the author’s structure, overlap can go to near zero. Where I keep it: a sentence or two of tail, only where a long section had to be cut by length anyway.
Tables are the sharper problem, and the failure is quiet. Split a table and the header row lands in one chunk while the numbers land in the next, so “4.5” and “18” arrive with nothing to say they are a torque value in Nm and a bolt count. For a machine builder, that is not an edge case — the prose is the packaging and the tables are the product. Three rules that hold up: never split a table across chunks; if one is genuinely too large, repeat the header row into every slice; and attach a one-line plain-language summary of what the table contains to the chunk you embed, because no one’s question ever looks like a row of numbers, and that summary is what the query actually matches.
Then the highest-return line of code in most chunking pipelines: prepend the heading path to the chunk’s text. A chunk reading “The limit is four weeks” is unusable — which limit? Prefixed with “Supply Agreement > §7 Warranty > 7.2 Notice of Defects”, it is both retrievable and citable. Note where it goes: into the text you embed, not only into a metadata field beside it. The embedding model sees the text and nothing else. Structured metadata earns its keep separately — source, version, date, page number, access scope — as pre-filters before the vector search and as the basis for citations, and access scope is what keeps a chunk out of an answer for someone not cleared to read the document. That side of the pipeline is covered in RAG pipelines explained.
How to Test Which Strategy Works on Your Documents
Everything above is a prior, not a verdict. Your documents get a vote, and the way most teams take that vote is useless: they read some answers and decide whether they look good. That confounds chunking with retrieval, ranking, prompting, and the model — five suspects and one clue.
Isolate it instead. Build a small gold set: 30 to 50 real questions, and — the part that matters — label the passage that actually contains the answer for each one. Then measure one number: how often the answer-bearing passage appears in the top k the retriever returns. Recall@k asks the generator nothing, which is exactly why it is the metric that isolates chunking. And a miss here is terminal: if the passage never surfaces, no prompt and no model recovers it. Run your candidate strategies against the same gold set and the argument becomes a decision.
Two things make this harder than it sounds. The questions have to come from people who did not build the system — support tickets, the search log, the mails the service desk gets — because questions invented by the project team are systematically too well-formed and quietly borrow the vocabulary of the documents. And labelling the answer passages needs someone who knows the material: a domain expert’s afternoon, not an engineer’s. That is the cheapest afternoon in the project and the one almost everyone skips — plenty of teams spend three weeks tuning splitters and zero days building the fifty questions that would have told them which splitter to keep.
Where Tippel Fits
Chunking rarely appears on a project plan. It shows up later, as “the assistant keeps missing things we know are in there” — and by then the fix means re-indexing the corpus and admitting nobody ever measured retrieval on its own. In the right order it is cheap; in the wrong order it is a rebuild.
If you have a RAG system that is nearly good enough, or documents you suspect are harder than they look, the AI Readiness Check is a timeboxed way to find out on your real files: what the parser recovers, which strategy wins on a gold set built from your questions, and whether the honest answer is “build it” or “not yet”. Fixed price, credited if you proceed. If you would rather talk it through first, get in touch.