Chunking Strategies for RAG: Getting Retrieval Right

A RAG system never searches your documents. It searches the pieces you cut them into. If an answer is split across two pieces, buried in a piece that is mostly about something else, or separated from the heading that says what it is about, retrieval can miss it no matter how good the embedding model is.

So chunking strategies for RAG are worth testing rather than copying. This guide tests five of them on the same documents and questions, and the result is more interesting than "smaller is better".

The choices

Libraries expose all of these. LangChain recommends starting with a recursive splitter that keeps paragraphs whole and only breaks them into sentences and words when they are too large, and also offers splitting by tokens, by characters and by document structure such as Markdown headers (LangChain: text splitters). LlamaIndex's examples use a sentence splitter with a chunk size of 1,024 and an overlap of 20 (LlamaIndex: node parsers). Defaults like these are starting points, not answers.

Why size matters at all

Two limits push towards smaller chunks.

The embedding model reads only so much. The model used here truncates anything beyond 256 word pieces (Hugging Face: all-MiniLM-L6-v2). Text past that point is not represented in the vector at all.

One vector has to stand for the whole chunk. A chunk about twenty topics produces a vector that sits between them and close to none.

Two forces push the other way. A chunk must contain enough to be understood on its own, and every chunk retrieved is text sent to the model.

A test

The documents are six public-domain Python Enhancement Proposals from peps.python.org, split into 148 sections at their headings. The script cuts them five ways, embeds every chunk, and asks the same 40 questions. A chunk counts as a hit only if it comes from the section that answers the question and contains a phrase that states the answer.

import json
import re

import numpy as np
from sentence_transformers import SentenceTransformer

sections = json.load(open("../../genai/pep_sections.json", encoding="utf-8"))
questions = json.load(open("../../genai/pep_questions.json", encoding="utf-8"))
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
question_vectors = model.encode([q["question"] for q in questions], normalize_embeddings=True)

too_long = sum(len(model.tokenizer(s["text"])["input_ids"]) > model.max_seq_length for s in sections)
print(f"{len(sections)} sections; {too_long} are longer than the {model.max_seq_length} tokens the embedding model reads")


def whole_sections(section):
    return [section["title"] + "\n" + section["text"]]


def fixed(size, overlap):
    def split(section):
        words = section["text"].split()
        return [" ".join(words[i:i + size]) for i in range(0, len(words), size - overlap)]
    return split


def paragraphs(with_title, limit=120):
    def split(section):
        chunks, current = [], []
        for paragraph in re.split(r"\n\s*\n", section["text"]):
            if current and len(" ".join(current + [paragraph]).split()) > limit:
                chunks.append("\n".join(current))
                current = []
            current.append(paragraph)
        chunks.append("\n".join(current))
        return [section["title"] + "\n" + c for c in chunks] if with_title else chunks
    return split


def flat(text):
    return " ".join(text.lower().split())


STRATEGIES = {"whole sections": whole_sections,
              "120 words, fixed": fixed(120, 0),
              "120 words, 30 overlapping": fixed(120, 30),
              "paragraphs up to 120 words": paragraphs(False),
              "paragraphs + section title": paragraphs(True)}
ranks = {}
print()
print("strategy                      chunks  words/chunk   answer in top 1   top 3   top 5")
for name, split in STRATEGIES.items():
    chunks = [(s["id"], c) for s in sections for c in split(s)]
    vectors = model.encode([c for _, c in chunks], normalize_embeddings=True)
    ranks[name] = []
    for q, qv in zip(questions, question_vectors):
        order = np.argsort(-(vectors @ qv))                 # a chunk counts if it holds the answer
        good = (pos for pos, i in enumerate(order, start=1)
                if chunks[i][0] == q["answer_in"] and flat(q["phrase"]) in flat(chunks[i][1]))
        ranks[name].append(next(good, None))
    found = [r for r in ranks[name] if r]
    words = np.mean([len(c.split()) for _, c in chunks])
    print(f"{name:28} {len(chunks):>7}  {words:>11.0f}   {sum(r == 1 for r in found):>15}   "
          f"{sum(r <= 3 for r in found):>5}   {sum(r <= 5 for r in found):>5}")

print()
print(f"{'rank of the chunk that holds the answer':50} whole sections   paragraphs   paragraphs + title")
for i in (21, 10, 38, 23):
    row = [ranks[n][i] or "never" for n in ("whole sections", "paragraphs up to 120 words", "paragraphs + section title")]
    print(f"{questions[i]['question'][:50]:50} {row[0]:>14}   {row[1]:>10}   {row[2]:>18}")

148 sections; 74 are longer than the 256 tokens the embedding model reads

strategy                      chunks  words/chunk   answer in top 1   top 3   top 5
whole sections                   148          212                21      28      33
120 words, fixed                 331           93                17      28      31
120 words, 30 overlapping        416           92                18      25      29
paragraphs up to 120 words       363           85                18      24      29
paragraphs + section title       363           89                21      27      33

rank of the chunk that holds the answer            whole sections   paragraphs   paragraphs + title
How should I check whether a variable is None?                 18            1                    1
Do I put spaces around the equals sign in a keywor              2           34                   39
How does the walrus operator's precedence compare               2        never                    1
Where does the phrase 'Readability counts' come fr              6           63                   73

What the totals say

• Half the sections are too long for the embedding model. 74 of 148 run past its 256-token limit, so part of each was never seen.
• Even so, whole sections did as well as anything. 21 answers ranked first and 33 in the top five.
• Fixed-size chunks did worse, and adding overlap did not help: 25 in the top three with overlap, against 28 without.
• Paragraphs on their own did worst, 24 in the top three.
• Adding the section title to each paragraph recovered nearly all of it: 21 first and 33 in the top five, the same as whole sections, with chunks less than half the size.

What the individual questions say

The totals hide large movements in both directions:

• "How should I check whether a variable is None?" went from rank 18 to rank 1. The rule sits deep in a section of more than 1,300 words, past the embedding model's limit. As a whole section it was invisible. As its own paragraph it was found at once.
• "Spaces around the equals sign in a keyword argument" went from 2 to 34. The answering paragraph is short and full of code examples. Without the sentences around it, it no longer looked like the answer to anything.
• The question about the walrus operator's precedence was never found without titles. The word "precedence" appears in the section's heading, not its paragraphs. Put the heading back on the chunk and it ranks first.
• "Readability counts" went from 6 to 63 and worse. The Zen of Python is a list of short lines that make sense together and little sense in pieces.

So splitting helped exactly where sections were long and mixed, and hurt where meaning depended on what surrounded the text.

What to take from it

1. Keep the heading with the chunk. In this test it was worth more than any choice of size or overlap. The same idea, taken further, is to add a short description of the chunk's source and subject before embedding it. Anthropic reports that this reduced failed retrievals by 49%, and by 67% with reranking (Anthropic: contextual retrieval).
2. Split by the document's structure before splitting by size. Sections, then paragraphs, then sentences.
3. Split long sections, keep short ones whole. Size only needs enforcing where a unit is too long for the embedding model.
4. Do not assume overlap helps. It adds chunks and storage. Here it changed little.
5. Treat lists, tables and code as special cases. They lose their meaning when separated from their introduction.
6. Check your embedding model's input limit, and make sure no chunk exceeds it.
7. Test with your own questions. The ranking of strategies here belongs to these documents. Yours may differ, which is the point of measuring. See retrieval evaluation.

Size and cost

Smaller chunks have one more advantage that the hit counts do not show. If three chunks are sent to the model with each question, then at the average sizes here that is about 640 words for whole sections and about 270 for titled paragraphs. Over thousands of questions that difference is real money and latency. See context windows explained.

A common pattern combines both: retrieve small chunks for precision, then pass the model the larger section each one came from, so it has the surrounding context to answer from.

Common mistakes

• Copying a chunk size from a tutorial without testing it on your documents.
• Cutting through the middle of sentences and tables with fixed-length splits.
• Throwing away headings.
• Chunks longer than the embedding model reads.
• Measuring retrieval by eye on a handful of favourite questions.
• Changing the chunking without re-embedding everything. Old and new chunks will not mix.

For the full pipeline these chunks feed, see RAG architecture explained. For combining embeddings with keyword search, which also helps with short or code-heavy chunks, see semantic search vs keyword search.

Document indexing and retrieval for RAG are part of the Vector Databases and Retrieval module of the Generative AI course.

Sources

• Text splitter integrations (LangChain documentation) — LangChain
• Node parser usage pattern (LlamaIndex documentation) — LlamaIndex
• Contextual Retrieval in AI Systems (Anthropic) — Anthropic
• all-MiniLM-L6-v2 model card (Hugging Face) — Sentence Transformers
• Python Enhancement Proposals (Python Software Foundation) — Python Software Foundation