Chunking at Scale: Where Retrieval Quality Is Won or Lost

Chunking Is Not a Preprocessing Step

Most teams treat chunking as plumbing. Someone picks 512 tokens with 50 tokens of overlap because a tutorial said so, ships it, and spends the next three months tuning prompts and swapping embedding models to fix problems that were created at ingestion. This is backwards. The chunk is the atomic unit of everything downstream: it is what gets embedded, what gets ranked, what enters the context window, and what the user eventually sees quoted back at them. A chunk that splits a clause from its condition, or a value from its column header, cannot be rescued by any model on the market.

At small scale the damage is invisible. On two thousand clean documents almost any chunking strategy produces acceptable results, because the corpus is small enough that the right document usually surfaces regardless. The strategy starts to matter somewhere around fifty thousand documents and becomes decisive past a few hundred thousand, because the number of plausible-looking near-matches grows with the corpus while the number of slots in your context window does not. At two million chunks, a query that would have had three competitors for a slot now has three hundred, and only precise chunk boundaries and good metadata keep the right one on top.

There is also a cost argument that gets ignored until the invoice arrives. Chunk size determines chunk count, chunk count determines embedding cost, index size, memory footprint and query latency. Halving chunk size doubles the vector count, which roughly doubles index memory and adds measurable latency to every approximate-nearest-neighbour search. On a corpus of a million documents, the difference between an average chunk of 300 tokens and one of 800 tokens is the difference between an index you can hold comfortably in memory on one machine and one that needs sharding, replication and an operations conversation. That decision is made at ingestion and is expensive to reverse.

Parse First: Layout-Aware Extraction and OCR Gates

Before you can chunk anything you have to turn a file into text, and this is where most real corpora lose the majority of their retrievable quality. A naive PDF text extractor reads a two-column page as a single stream, interleaving the left and right columns line by line and producing text that is grammatically impossible. It drops headers and footers into the body, so a page number and a document title appear mid-sentence. It renders footnotes inline, merges hyphenated line breaks incorrectly, and silently returns an empty string for pages that are actually images. Every one of those failures poisons the chunk before any chunking logic runs.

Layout-aware extraction is the fix, and it means using a parser that recovers reading order, block types and the document hierarchy rather than a raw character stream. What you want out of the parser is a structured tree: title, section headings with their nesting level, paragraphs, list items, tables as tables, figures with captions, and footnotes attached to their anchors. That tree is what makes structure-aware chunking possible later. Budget real time for this: on a mixed corpus, parsing and extraction work routinely accounts for 30 to 50 percent of the total ingestion engineering effort.

Scanned documents need a quality gate, not just an OCR pass. Run OCR, then score every page and route it. We use a combination of the engine's own mean character confidence, the proportion of tokens that appear in a dictionary for the expected language, and simple structural sanity checks such as whether line lengths are plausible. Pages that score well go straight through. Pages in a middle band get flagged and indexed with a low-confidence marker so that anything retrieved from them can be presented with a caveat. Pages below the floor should not be silently indexed, because a page of OCR garbage is worse than a missing page: it produces confident nonsense rather than an honest gap.

Decide explicitly what happens to the pages that fail. The options are a second OCR pass with a different engine or preprocessing such as deskewing and contrast normalisation, a re-scan request routed to whoever owns the physical document, or an entry in an exception register that says this page is not available to the system. In corpora of scanned Turkish documents from the 1990s and 2000s we routinely see 3 to 8 percent of pages fail a reasonable quality gate. Publishing that number to stakeholders early is far better than letting them discover coverage gaps through wrong answers six months later.

Tables, Figures and What PDFs Do to You

Tables are the single most common cause of confidently wrong answers in document RAG, and the reason is mechanical. A table flattened to text loses the association between a cell and its column header. A row that reads 12.500 in a flattened stream might be a price, a quantity, a threshold or a year, and the model will guess based on surrounding prose. Worse, tables frequently span pages, so the header row appears once on page four and the continuation rows on page five arrive with no header at all. A chunker that splits mid-table produces fragments that are individually meaningless and individually retrievable, which is the worst combination.

The workable pattern is to treat a table as an object rather than as text. Extract it as structured rows, then render it into the chunk in a format that preserves the header association, such as a markdown-style or key-value serialisation where each row repeats its column names. Keep the table whole in one chunk when it fits, and when it does not, split by row groups and repeat the header block in every fragment. Attach a caption and the surrounding paragraph as a prefix, because tables are almost never self-describing, and add a metadata flag marking the chunk as tabular so you can measure retrieval quality on tables separately from prose.

Figures need a decision rather than a default. An image with a caption should at minimum be indexed by its caption plus surrounding text. For diagrams, charts and schematics that carry information not present in the prose, a vision model can generate a text description at ingestion, which then gets embedded like any other chunk. This costs real money at scale, so gate it: describe figures only in the document classes where figures actually carry meaning, and skip decorative images entirely. Whatever you choose, record it, because a year later somebody will ask why the system cannot answer questions about the process diagram on page nine.

Four Chunking Strategies and When Each Wins

Fixed-size chunking splits on a token count with an overlap, ignoring structure entirely. It is fast, trivially parallel, predictable in cost and completely indifferent to meaning. It is the right choice for genuinely unstructured text such as chat transcripts, free-text notes and OCR output too noisy to trust structurally. It is the wrong choice for anything with headings, clauses or tables, which is most enterprise content. Recursive chunking is the sensible default: split on a priority list of separators, paragraph breaks first, then sentence boundaries, then whitespace, packing text until the size limit is reached. It costs almost nothing extra and avoids the most egregious mid-sentence cuts.

Structure-aware chunking uses the document tree from your parser and splits on real boundaries: an article, a clause, a section, a numbered item. It produces chunks that a human would recognise as units, which makes citations trustworthy and makes debugging comprehensible. It is clearly the best option for regulations, contracts, standards, policies and technical manuals, and it is the approach behind the article-level handling described in RAG on Turkish legal data. The cost is that it depends entirely on parsing quality, and it produces highly variable chunk sizes, so you need a merge pass for tiny sections and a split pass for enormous ones.

Semantic chunking computes embeddings for successive sentences and cuts where the similarity between neighbours drops below a threshold, on the theory that a topic shift is a natural boundary. It sounds compelling and it sometimes helps on long unstructured prose. In practice it is expensive, because you embed the corpus twice, it is sensitive to a threshold nobody knows how to set, and on structured documents it usually underperforms simply splitting on the headings that the author already wrote for you. Our position is to reach for it only when the content genuinely has no usable structure and recursive chunking has been measured and found wanting.

In production these are not exclusive. A realistic pipeline routes by document class: structure-aware for contracts and regulations, table-object handling for spreadsheets and financial reports, recursive for web pages and internal wiki content, fixed-size for transcripts, with a merge-and-split normalisation pass at the end so that the final chunk-size distribution is controlled regardless of which path produced it. The routing decision comes from the file type plus a lightweight classifier over the first page, and it is worth the two or three days it takes to build. Document the routing table itself, because it is the first thing you will consult when a bad answer needs explaining.

Chunk Size and Overlap Are Tunables, Not Constants

There is no correct chunk size, only a size that is correct for your corpus, your question distribution and your embedding model. The useful ranges to search are roughly 200 to 400 tokens for dense factual lookup where the answer sits in one or two sentences, 400 to 800 tokens for general document question answering, and 800 to 1,500 tokens for narrative or analytical content where the answer requires an argument rather than a fact. Overlap of 10 to 20 percent of chunk size is the usual band. Anything above 25 percent mostly inflates your index and your embedding bill without measurably improving recall.

Treat these as a parameter sweep, not a decision. Build the evaluation set described in RAG evaluation metrics first, then index the same corpus at three or four size and overlap combinations and measure recall@k and context recall for each. On a corpus of a hundred thousand documents this is a day of compute and a few hundred lira of embedding spend, and it routinely moves recall by five to fifteen points. That is a larger improvement than most model upgrades deliver, for a fraction of the effort, and it is the single most common piece of free quality we find in systems that were built from a tutorial default.

Watch the distribution, not just the mean. A corpus with a mean chunk size of 500 tokens can still be pathological if a quarter of its chunks are under 50 tokens, which happens constantly with structure-aware splitting over documents full of short headings and one-line clauses. Very short chunks embed poorly because there is not enough signal in them, and they crowd out substantive chunks in the ranking because short text can score high similarity on a narrow query. Set a floor of roughly 80 to 120 tokens and merge anything below it into its neighbour, and set a ceiling that forces a split with the section path preserved on both halves.

Small-to-Big: Contextual and Parent-Document Retrieval

There is a real tension between the chunk size that retrieves well and the chunk size that answers well. Small chunks have concentrated meaning and therefore match queries precisely, but they often lack the surrounding context a model needs to produce a complete answer. Large chunks answer well but retrieve badly, because their embedding is an average of several topics and matches nothing sharply. Small-to-big retrieval resolves this by decoupling the two: index small chunks for matching, but when a small chunk is retrieved, return its larger parent to the model.

The parent-document pattern is the straightforward implementation. Store each document as a hierarchy: the whole document, sections, and child chunks of 200 to 400 tokens. Embed only the children. At query time, retrieve children, map them to parents, deduplicate parents that were hit by more than one child, and pass the parent sections into the context window. The main thing to watch is context budget, because a single parent section can be several thousand tokens and three of them will crowd out everything else. Cap the number of parents and fall back to the child chunk when a parent exceeds a size limit.

Contextual chunk augmentation attacks the same problem from the other side. Before embedding, prepend a short generated or templated context line to each chunk that situates it: the document title, the section path, the effective date and a one-sentence summary of what this chunk is about relative to the whole document. This makes an isolated clause retrievable by queries that name the document or the topic without repeating the clause's own vocabulary. Generating those context lines with a language model costs one cheap call per chunk at ingestion, which on a million chunks is a real budget line, so a templated version built from the parsed section path captures much of the benefit at effectively zero cost.

Metadata Design: Often Worth More Than the Embedding

The most under-invested part of a RAG pipeline is the metadata schema, and it is frequently the part with the highest return. A user asking about the current expenses policy does not want a semantically excellent match against the 2019 edition. No embedding model can know which version is current; only a field can. Design the schema before ingestion and make it mandatory: source system and document identifier, document type, title, section path, page or article number, effective date and expiry date, version identifier and supersession pointer, language, confidence flags from the OCR gate, and access-control labels.

Access-control labels deserve emphasis because retrofitting them is painful and dangerous. Every chunk should carry the identifiers of the groups permitted to see it, and filtering must happen inside the retrieval query rather than after ranking. Filtering after the fact silently degrades quality, because a user without access to three of the top five results receives an answer built from the remaining two without any indication that the retrieval was gutted. It is also a leak risk the moment anyone logs the unfiltered candidate list. Get the labels into the index from day one and test them with a permission-differential evaluation set where the same question is asked by two users with different rights.

Metadata pays off in three distinct ways. It enables hard filters that eliminate whole classes of wrong answers, such as restricting to documents effective on the date the user asked about. It enables boosting, where a recent or authoritative document gets a score multiplier during fusion, which is discussed further in hybrid search and reranking. And it makes citations useful, because a citation that names the document, the section and the effective date can be verified by a human in seconds, while a citation that says chunk 48213 cannot. If you have limited engineering time, improving metadata beats changing the embedding model more often than not.

Deduplication, Versions and Superseded Documents

Large corpora are full of near-duplicates, and they are corrosive in a way that exact duplicates are not. The same policy exists as a Word file, a PDF export and an intranet page. A contract template appears in ninety client folders with only the names changed. A regulation is quoted in full inside twelve internal memos. When a query matches this content, the top five results are five copies of the same passage, retrieval diversity collapses, and the model receives one fact repeated five times instead of five facts. The user gets a confident answer built on a single source that appeared to have overwhelming support.

Deduplicate in two stages. Exact duplicates go first via a content hash computed after normalisation, which is cheap and catches the multiple-export case. Near-duplicates need similarity detection: MinHash or SimHash over shingles is the scalable option at tens of millions of chunks, with a cosine-similarity check over embeddings above roughly 0.95 as a more expensive refinement on smaller sets. Rather than deleting matches, cluster them, elect a canonical chunk by source authority and recency, and keep the others linked to the canonical one so you can still report where else the text appears. Then apply diversity at query time as well, capping how many chunks from a single document may occupy the final context.

Versioning is the harder problem and it is where document RAG most often produces answers that are wrong in a way nobody notices. Each document needs an effective date, an optional expiry date and an explicit supersession pointer to the version that replaced it. The retrieval layer then needs a default temporal policy: usually restrict to currently effective documents, with an explicit as-of mode for questions that are genuinely historical. Superseded chunks should be marked rather than deleted, because audit questions about what the rule used to be are legitimate and common, particularly in regulated industries.

Turkish-Language Specifics and Mixed Corpora

Turkish is agglutinative, so a single stem generates a very large number of surface forms, and lexical retrieval that treats those forms as unrelated tokens will simply miss documents. Dense retrieval handles this reasonably well when the embedding model has genuinely seen Turkish, but the lexical half of any hybrid system needs help: a Turkish analyser with proper stemming or lemmatisation, correct handling of the dotted and dotless i, and awareness that lowercasing Turkish text with an English locale corrupts it. This is a configuration detail with outsized consequences, and it is one of the most common defects we find when reviewing an existing Turkish search index.

Sentence segmentation is the second trap. Turkish uses periods in ordinal numbers, in dates written as 12.03.2026, in abbreviations such as vb. and Av., and in legal references, so a naive sentence splitter fragments text at exactly the points where meaning is densest. Since recursive and semantic chunking both depend on sentence boundaries, a bad segmenter propagates errors straight into chunk boundaries. Use a segmenter with a Turkish abbreviation list and numeric-context rules, and validate it by sampling a few hundred split points manually before you trust it across a million documents.

Mixed corpora are the normal case rather than the exception. Turkish enterprises routinely hold documents that are Turkish prose with English technical terms, English contracts with Turkish annexes, and tables whose headers are English while the values are Turkish. Detect language per chunk rather than per document, store it as metadata, and verify that your embedding model handles both languages in a shared space rather than clustering by language, which some models do and which quietly destroys cross-lingual retrieval. Test explicitly for the case that matters most in practice: a Turkish question that must retrieve an English passage, and the reverse. The behaviour of your vector database and semantic search layer on that test is worth knowing before launch, not after.

Incremental Re-Indexing, and How We Run This at Scale

Corpora change, and a pipeline that can only rebuild from scratch will stop being rebuilt. Design for incremental updates from the start: a stable document identifier that survives re-upload, a content hash per document and per chunk so unchanged content is skipped, and a change feed that carries creates, updates and deletes rather than requiring a full directory scan. When a document changes, re-parse it, re-chunk it, diff the chunk hashes, and upsert only what actually moved. On a typical enterprise corpus a daily delta touches well under one percent of documents, so incremental ingestion turns an eight-hour rebuild into a job that finishes in minutes.

Some changes still force a full re-embed, and you should know which. Changing the embedding model, the chunk size, the overlap or the contextual prefix template invalidates every vector in the index. Plan for it with a blue-green pattern: build the new index alongside the old one, run the evaluation suite against both, switch reads over when the new index wins, and keep the old one for a rollback window. On a million-document corpus a full re-embed is typically a few hours of wall-clock time with reasonable parallelism, so the constraint is usually cost and rate limits rather than compute.

Our own practice starts by measuring the parser before anyone argues about chunk size. On a new corpus we sample 50 to 100 pages across document classes, extract them, and read the output by hand, which sounds primitive and finds more defects per hour than any automated check we have tried. Then we build the evaluation set, then we sweep chunk parameters against it. Across the document-heavy work behind caseon.ai and DiligenceAI, the pattern has been consistent: parsing and metadata work produced larger and more durable quality gains than model changes, and the systems that aged well were the ones where superseded versions were marked rather than deleted.

The trade-off worth stating plainly is that this work is unglamorous and it is where the schedule actually goes. On a 50,000 to 200,000 document corpus we typically spend two to four weeks on parsing, extraction, OCR gating and chunk-parameter tuning before the retrieval quality curve starts to flatten. Teams that skip it do not save the time; they spend it later on prompt engineering that cannot work, and on the long-context temptation examined in long context versus RAG. If you are building on a large document estate, our RAG and semantic search practice treats ingestion as the main engineering problem, because at scale it is.