Why Chunking Decides What Retrieval Can Ever Find
A retriever cannot return a passage that does not exist as a chunk. The embedding model, the reranker, the prompt and the model itself therefore only choose from what chunking laid out. If the one sentence that answers a question sits in a chunk whose other 700 tokens are about something else, the vector for that chunk averages both topics and matches neither precisely.
On most business corpora, the winning strategy therefore follows the structure the document already has and leaves chunk size alone. The elaborate methods usually compensate for a parser that destroyed that structure before the chunker saw it. A chunking debate therefore hides a parsing problem more often than a chunking problem.
Chunking also demands more care than an ordinary parameter, because the decision is hard to take back. Move the boundaries and you invalidate every vector in the index, re-embed the whole corpus and repeat the evaluation. On a few thousand documents that costs an afternoon; on a machine builder’s manual archive it costs a batch window and a bill.
Three Chunking Strategies, Ranked
The discussion covers three methods: fixed-size, semantic and structure-aware splitting. They do not perform alike, and the semantic one attracts far more attention than its results justify.
Fixed-size splitting: the baseline you have to beat
The splitter cuts every N tokens and adds a little overlap. The common form is LangChain’s RecursiveCharacterTextSplitter, which by default breaks on paragraphs first, then line breaks, then spaces and finally single characters. The method runs deterministically and fast, costs nothing and is easy to debug, because “why is this chunk here?” always has a readable answer. It suits corpora without structure, such as call transcripts, chat logs, forum threads and OCR from handwritten shop-floor notes. Its failures show: it severs a numbered clause from its number and cuts a procedure between step 4 and step 5.
Semantic chunking: the one I would not start with
The method embeds each sentence, measures the distance to the next and cuts where the distance jumps. In principle that finds topic boundaries no formatting marks. In practice, on typical company corpora, it mostly rediscovers the paragraph breaks the document already supplies for free as characters in the text. In exchange, ingest costs one embedding call per sentence, and across an archive of 900-page manuals that adds up to a bill and a batch window.
The heavier drawback is that the boundaries depend on the embedding model, which makes the method non-deterministic in that sense: a model upgrade silently reshapes the whole index. Anyone investigating a bad answer can then no longer answer “why does this chunk start here?”. The method suits long unstructured prose in which topics drift with no formatting cue, such as interview notes, and I would not rule it out there. I would not make it the default, and never before a test set proves it earns its cost.
Structure-aware splitting: use the skeleton the author wrote
Documents that matter to a business are almost never flat text. A supply agreement has numbered paragraphs, a maintenance manual has chapters, sections and numbered procedures, and lawyers worked for a long time on the clause structure of a regional insurer’s policy conditions. Someone has therefore already segmented the text semantically, by hand and with headings, and stored the result in the document. Split on those boundaries and you take over that work, whether with a Markdown heading splitter, a pass over the HTML headings or the outline of a DOCX.
The work therefore sits upstream of the chunker, because structure-aware chunking needs a parser that preserved the structure. Convert the documents to a structured intermediate format such as Markdown or HTML, with a tool built for the job: Docling, Unstructured or Azure Document Intelligence. A team that dumps a PDF to plain text instead loses the outline in the first step and then needs semantic chunking to reconstruct what it threw away.
Structure-aware chunking is fixed-size chunking, except that the author sets the boundaries and not a modulo operation. The two methods therefore do not compete. You split on structure, and where a section still runs past your ceiling, you split it by length.
Stop Choosing a Chunk Size
The size debate over 300 or 1,000 tokens assumes that one number must serve two jobs with opposite demands. Retrieval wants small chunks, because an embedding is a single fixed-length vector that blurs as a chunk holds more distinct ideas. Generation wants big chunks, because the model needs the definition two paragraphs up and the exception in the sentence below, or it answers confidently from half the facts.
You do not have to choose, because the chunk you retrieve need not be the chunk you send. Index the small unit, store a pointer to its parent section and expand a hit to that parent. LlamaIndex ships this as auto-merging retrieval, LangChain as ParentDocumentRetriever (in the langchain-classic package since version 1.0). If you prefer your own code to the dependency, a few dozen lines suffice.
This costs tokens, because five small hits can expand into three parent sections; deduplicate the parents and cap the context budget. In return the parameter the size debate is about drops out, and usually no other single change improves a mediocre RAG system as reliably.
Overlap, Tables, and the Line That Rescues a Chunk
Overlap of 10 to 20 percent counts as the default setting. It covers boundaries you do not trust, and in exchange it costs index size and top-k slots. When the same sentence ranks first and second, one idea takes two of five places in the context. When the boundaries follow the author’s structure, overlap can go towards zero. I keep a sentence or two of tail only where I had to cut a long section by length anyway.
Tables weigh more, because nobody notices the failure. When the splitter cuts a table, the header row lands in one chunk and the numbers in the next, and “4.5” and “18” arrive with nothing to say they are a torque value in Nm and a bolt count. For a supplier or machine builder that is no edge case, because its documents carry their content in the tables and not in the prose. Three rules have proved themselves. Never split a table across chunks. If it is too large, repeat the header row in every slice. Also attach a one-line plain-language summary of the table’s contents to the chunk you embed, because no question looks like a row of numbers, and the query matches that summary.
In almost every chunking pipeline, one line of code brings the highest return: it prepends the heading path to the chunk text. A chunk reading “The limit is four weeks” does not say which limit and is therefore unusable. Prefixed with “Supply Agreement > §7 Warranty > 7.2 Notice of Defects”, it can be retrieved and cited. The path belongs in the text you embed and not only in a metadata field beside it, because the embedding model sees only the text. Structured metadata such as source, version, date, page number and access scope serves separately as a pre-filter before the vector search and as the basis for citations. Access scope keeps a chunk out of an answer for someone who may not see the document. RAG pipelines explained covers that side of the pipeline in detail.
How to Test Which Strategy Works on Your Documents
Everything above is a prior and not a verdict, because your documents decide the strategy. Most teams read a few answers and judge them good. That says nothing about chunking, because the answer depends just as much on retrieval, ranking, prompt and model.
Isolate chunking instead. Build a small gold set of 30 to 50 real questions and label, for each, the passage that contains the answer; the value of the set depends on that labelling. Then measure one number, namely how often the passage with the answer appears in the retriever’s top k. Recall@k asks the generator nothing and therefore isolates chunking. A miss here is final, because no prompt and no model recovers what never surfaces. Run your candidates against the same gold set, and the numbers settle the discussion.
Two things make this harder. The questions have to come from people who did not build the system, meaning from support tickets, the search log and the mails the service desk receives. Questions the project team invents are systematically too well-formed and quietly adopt the vocabulary of the documents. Someone who knows the material labels the answer passages, and that costs a domain expert an afternoon and not an engineer. That afternoon is the cheapest in the whole project, and almost everyone skips it. A team that spends weeks tuning splitters and not one day on the fifty questions never learns which splitter to keep.