Chunking Strategies for RAG: Why Fixed-Size Splits Keep Failing You

Most RAG systems don't fail at retrieval. They fail at chunking. By the time your embeddings are in Pinecone or pgvector, the damage is already done — you sliced a contract mid-clause, split a code block from its explanation, or shredded a table into unreadable rows.

We've rebuilt enough retrieval pipelines to know the pattern: teams reach for RecursiveCharacterTextSplitter, ship it, and then spend three months tuning reranking to compensate for bad chunks. This is a walk through the strategies that actually move retrieval quality, when each one earns its complexity, and the tradeoffs we've hit in production.

Why fixed-size chunking is the wrong default

The standard advice — split every 512 or 1000 tokens with some overlap — treats every document like it's an English essay. It isn't. A support KB article, a PDF earnings report, a Markdown API reference, and a Slack thread all have different structural signals, and fixed-size splitting ignores every single one of them.

The symptoms show up downstream:

• The model cites a chunk that starts mid-sentence and hallucinates the missing context.
• Two adjacent chunks return for the same query because the overlap window duplicated the useful bit.
• Table rows retrieve without their headers, so numbers appear without units.
• Code samples come back stripped of the function signature that named them.

Overlap helps a little. It also inflates your index size by 15 – 30% and increases the odds of near-duplicate retrievals crowding out diverse context. It's a patch, not a fix.

Match the split to the document structure

The first question we ask on any new RAG project isn't "what chunk size?" It's "what does this document look like?" Structure-aware splitting almost always beats a smarter embedding model.

Markdown and HTML: split on headings first

For docs, wikis, and rendered HTML, headings are free semantic boundaries. Split on ## and ### first, then only fall back to size-based splitting if a section is too large for your embedding model's context.

import re
from dataclasses import dataclass

@dataclass
class Chunk:
    text: str
    heading_path: list[str]
    source: str

def split_markdown_by_heading(md: str, source: str, max_chars: int = 2000):
    sections = re.split(r'(^#{1,3} .+$)', md, flags=re.MULTILINE)
    path, chunks, buf = [], [], ''

    for part in sections:
        if part.startswith('#'):
            level = len(part) - len(part.lstrip('#'))
            title = part.strip('# ').strip()
            path = path[:level - 1] + [title]
        else:
            buf = part.strip()
            if not buf:
                continue
            # Only fall back to size split if the section is too big
            if len(buf) <= max_chars:
                chunks.append(Chunk(buf, path.copy(), source))
            else:
                for i in range(0, len(buf), max_chars):
                    chunks.append(Chunk(buf[i:i+max_chars], path.copy(), source))
    return chunks

The crucial bit isn't the split — it's carrying heading_path along as metadata. When you retrieve a chunk, you prepend "From: Billing > Refunds > Partial refunds\n\n" before sending it to the model. That single change usually beats a week of prompt tweaking.

PDFs: layout is the signal

PDFs are the hardest case. Naive text extraction destroys tables and multi-column layouts. We reach for tools that preserve layout — unstructured, pdfplumber, or vendor APIs like AWS Textract or Google Document AI when tables actually matter. Chunk boundaries follow the visual structure: sections, tables (kept whole), figures with their captions.

If a table is too big for one chunk, split it by row groups and repeat the header on every chunk. Same principle for code — repeat the enclosing function signature.

Code: use the AST

For code repos, splitting by lines is malpractice. Use tree-sitter or the language's own parser and chunk by function or class. Each chunk carries the file path, the language, and imports as metadata so the model knows what it's looking at.

Semantic chunking: worth it, but not always

Semantic chunking — embedding each sentence and starting a new chunk when the cosine distance to the previous sentence spikes — has become fashionable. It's a real technique, first popularised by Greg Kamradt and now available in LangChain and LlamaIndex.

It works well when:

• Your documents are prose with no reliable structural markers (transcripts, legal opinions, research prose).
• Topics drift within a single section and headings don't reflect it.

It's overkill when:

• You already have good structural signals (Markdown, well-formed HTML).
• Your corpus is huge — you're paying embedding cost twice, once at chunking time and once for the final index.
• Latency in the ingestion pipeline matters.

In our experience, on structured content, structural chunking plus a decent reranker beats semantic chunking on retrieval quality and costs less. On unstructured prose, semantic chunking earns its keep.

Hierarchical chunking: the pattern we default to now

The single upgrade that has paid off most consistently is hierarchical or "parent-child" chunking. The idea: index small chunks for precise retrieval, but return larger parent chunks to the model for context.

A typical setup:

1. Split each document into ~2000-character parent chunks along structural boundaries.
2. Split each parent into ~400-character child chunks.
3. Embed and index the child chunks only.
4. At query time, retrieve top-k child chunks, deduplicate by parent, and send the parents to the LLM.

Why it works: small chunks embed cleanly (embeddings degrade on long, topic-mixed text), but small chunks alone give the LLM tunnel vision. The parent gives it room to reason.

def build_hierarchical(chunks, parent_size=2000, child_size=400):
    parents, children = [], []
    for parent_id, section in enumerate(chunks):
        parents.append({'id': parent_id, 'text': section.text, 'meta': section.heading_path})
        for i in range(0, len(section.text), child_size):
            children.append({
                'parent_id': parent_id,
                'text': section.text[i:i+child_size],
                'meta': section.heading_path,
            })
    return parents, children

def retrieve(query, index, parents, k=8):
    child_hits = index.search(query, top_k=k * 3)
    seen, results = set(), []
    for hit in child_hits:
        pid = hit['parent_id']
        if pid in seen:
            continue
        seen.add(pid)
        results.append(parents[pid])
        if len(results) == k:
            break
    return results

Both LlamaIndex (AutoMergingRetriever) and LangChain (ParentDocumentRetriever) ship variants of this. Roll your own if their abstractions get in the way — the logic is 40 lines.

Contextual retrieval: prepend a summary to every chunk

Anthropic published a technique in September 2024 they call contextual retrieval: before indexing each chunk, ask an LLM to write a one-sentence summary of how that chunk fits into the whole document, and prepend it to the chunk. Their reported reduction in retrieval failure was significant, and prompt caching makes it affordable.

We've used it on customer-facing knowledge bases where the same product name means different things in different sections. It genuinely helps disambiguation. The tradeoff is real cost at ingestion time — you're running an LLM call per chunk — but with prompt caching against the source document, most of that cost collapses.

If you go this route, batch aggressively and cache the document content, not per-chunk. Anthropic's docs cover the caching model in detail.

Evals: don't tune chunking by vibes

If you change chunk size and it feels better, you have learned nothing. Build a small retrieval eval — 50 to 200 real questions with the document IDs that should be retrieved — and measure recall@k and MRR whenever you touch the pipeline. This is the same discipline we've written about for evals elsewhere, applied at the retrieval layer specifically.

A few metrics we track:

• Recall@10: does the right document appear in the top 10? If this is under ~85%, no amount of reranking saves you.
• Chunk redundancy: how many of the top-k chunks share more than 50% content? High numbers mean overlap or hierarchical dedup is broken.
• Answer groundedness (separate LLM eval): does the final answer actually cite retrieved content?

Without these, you're guessing. With them, you can defend chunk-size decisions in a code review.

Where we'd start

On a fresh RAG project, we default to this stack: structural chunking driven by document type, hierarchical parent-child indexing, heading path stored as metadata and prepended at retrieval time, and a 100-question retrieval eval before we touch anything else. Semantic chunking and contextual retrieval come in as targeted upgrades once the eval identifies where recall is bleeding — not as day-one complexity.

If your RAG system is underperforming today, don't rewrite the retriever. Print out ten of your worst chunks. Nine times out of ten, the answer is looking back at you. If you want a second pair of eyes on a pipeline that's stuck, our AI engineering team does this work under NDA.