Chunking

A chunk is a passage cut from a document, typically a few hundred words. Documents are chunked for three reasons:

• Embedding models have a maximum input length and ignore anything past it.
• One vector for a whole 40-page manual is a blurry average of every topic in it, close to nothing in particular.
• The model should be handed the relevant passage, not the whole manual, to keep the prompt small and focused.

The chunk is the unit of everything downstream: it is what gets embedded, what gets retrieved, what gets cited and what the model reads. A fact that was split across two chunks, or buried in a chunk about five other things, is effectively lost.

Chunk size is usually measured in tokens (a token is a word-piece; 100 tokens is roughly 75 English words).

• Small chunks (100 to 300 tokens) have a precise, focused meaning, so they match questions well, but they may lack the surrounding context needed to answer.
• Large chunks (800 to 1,500 tokens) carry full context, but their vector is a blur of several topics, so they match less precisely, and they fill the prompt faster.

There is no universally right size; there is a right size for your documents and questions.

• Short factual questions over FAQs favour small chunks.
• Questions that need reasoning over a long explanation favour larger ones.
• A common starting point is 300 to 500 tokens, then measuring retrieval quality at two or three sizes on real questions.

Overlap means each chunk repeats the last part of the previous one.

• With a size of 500 tokens and an overlap of 100, chunk 1 covers tokens 0 to 500, chunk 2 covers 400 to 900, and so on.
• The window moves forward by size − overlap each time.

A cut can land in the middle of an idea: the rule in one chunk, its exception in the next.

• Overlap means a sentence near a boundary appears whole in at least one chunk.
• The cost is storage and some repetition in results.
• An overlap of 10 to 20 percent of the chunk size is the usual range.

• Fixed-size splitting cuts every N tokens, wherever that lands, even mid-sentence.
• Recursive splitting is the sensible default: it tries to cut at the biggest natural boundary first (a blank line between paragraphs), and only if a piece is still too long does it fall back to the next boundary (a line break, then a sentence end, then a space).

Recursive splitting keeps paragraphs and sentences whole wherever the size limit allows, at no extra cost. It is what most libraries offer as their standard text splitter, and it is a reasonable baseline to measure everything fancier against.

Structure-aware chunking cuts along the document's own outline: one chunk per section or sub-section, using the headings the parser preserved.

• Markdown is split at # headings, HTML at heading tags, a contract at numbered clauses, code at functions and classes.
• Sections that are too long are then split further; very short ones are merged with a neighbour.

Authors already grouped related sentences under a heading, so a section is usually a complete thought. Each chunk can also carry its heading path ("Refunds > Late orders") as metadata, which improves search, gives a precise citation and tells the model where the passage sits.

Semantic chunking looks for where the topic changes. It embeds each sentence, compares every sentence with the next, and starts a new chunk wherever the similarity drops sharply, which signals a change of subject.

It helps with text that has no headings to cut on: transcripts, long emails, chat logs.

• The chunks follow the content instead of a token count.
• It costs an embedding call per sentence at indexing time and produces chunks of uneven length, so it needs a maximum size as a guard.

A way to escape the size trade-off.

• Index small child chunks (a few sentences) for searching, and keep a link from each to its larger parent (the whole section).
• Search matches against the precise child vectors, then the system hands the model the parent text.
• A close relative, sentence-window retrieval, matches a single sentence and returns it with the few sentences on either side.

It gives you both halves of the trade-off: the sharp matching of small chunks and the full context of large ones. It is one of the most reliable upgrades over plain chunking, and a common interview answer to 'how would you improve retrieval without changing the model?'.

A chunk read on its own often does not say what it is about.

• "The fee is refunded within 5 days": which fee, which policy?
• A contextual header fixes this by putting a line of context in front of the chunk before it is embedded: the document title and section path, or a one-sentence description written by a language model that has seen the whole document.

The added line gives the vector the missing subject, so the chunk can be found by questions that name it, and gives the model the missing subject when it reads the chunk. It is cheap, and it reliably improves retrieval on documents full of 'it', 'this plan' and 'the above'.

Tables and code break under text splitters. A table cut in the middle leaves rows without their column headings. The fixes for tables:

• keep a small table as one chunk;
• split a large one by rows and repeat the header row in every chunk;
• or turn each row into a sentence.

Code should be cut at function and class boundaries, with the file path and, for a method, its class name attached.

A chunk has to make sense by itself.

• | 199 and above | Free | means nothing without the header that says the columns are order value and delivery fee.
• Half a function is equally unreadable to both the search and the model.

A workable order of attack:

• Start with structure-aware chunking if the documents have headings, or recursive splitting if they do not, at 300 to 500 tokens with 10 to 15 percent overlap.
• Add contextual headers.
• If answers lack context, move to parent-child.
• Reach for semantic chunking only for unstructured text.

Decide each step by measuring retrieval on a fixed set of test questions.

Interviewers ask 'how would you chunk this?' to see whether you reason from the documents and the questions, not from a favourite number. The strong answer names the document type, proposes a default, says what you would measure, and says what you would change if the measurement were poor.