Structure-Aware Chunker With Overlap

Chunking is where most RAG systems are quietly lost. Retrieval cannot return what chunking destroyed.

Implement chunk_text(text, max_tokens, overlap_tokens):

• Split text into sentences on ., ! or ? followed by whitespace
• Pack whole sentences into chunks of at most max_tokens tokens (a token is a whitespace-separated word)
• Consecutive chunks overlap by approximately overlap_tokens tokens, carrying the trailing sentences of the previous chunk into the next
• Never split a sentence across chunks
• A single sentence longer than max_tokens becomes its own chunk, oversized

Return a list of chunk strings.

Overlap exists because the sentence that answers a question is often the one after the sentence that matches it. Without overlap, a boundary falling between them makes the answer unretrievable.