Document Chunking: Slicing Text for LLMs

WHY IT EXISTS Embedding models can only process a fixed amount of text at once, known as their context window. Feeding them a document larger than this window means information gets cut off and lost. Furthermore, for search to be effective, the units of text being searched must be semantically meaningful on their own.

THE MENTAL MODEL Chunking is like preparing a book summary chapter by chapter, not sentence by sentence. Each chunk must be a self-contained idea. The rule of thumb is: if a chunk of text makes sense to a human without the surrounding context, it will make sense to the language model. A chunk that is just a few random sentences is useless.

HOW IT WORKS A large document is processed by a chunking strategy, which splits the text into smaller segments. These chunks are then individually converted into vector embeddings and stored in a vector database. The goal is to create chunks that are big enough to contain meaningful, context-rich information, but small enough to be processed efficiently and fit within model context windows.

WHEN TO USE IT Chunking is an essential preprocessing step for nearly any application that retrieves information from a knowledge base to use with an LLM. It is fundamental to Retrieval-Augmented Generation (RAG), where retrieved chunks provide grounding context for the model's response. It is also critical for semantic search, where user queries are compared against the chunks in a vector database to find the most relevant information.

WHEN NOT TO USE IT Even with models that have very large context windows (e.g., 200k tokens), chunking is still highly recommended. Feeding a whole large document into a model increases latency and cost. More importantly, models suffer from the "lost-in-the-middle" problem, where they may ignore relevant information buried deep inside a long document. Chunking ensures only the most relevant information is passed to the model, improving quality and efficiency.

ONE CANONICAL EXAMPLE Imagine building a Q&A bot over your company's internal documentation. Instead of trying to stuff a 50-page PDF into a prompt, you would first chunk it into logical sections (e.g., by paragraph or subsection). When a user asks about the remote work policy, the system performs a semantic search to find the most relevant chunk(s) and provides only those to the LLM. This is faster, cheaper, and yields a more accurate answer.