Semantic Chunking Is Overrated: What Actually Works for Production RAG

Every few months a new chunking strategy makes the rounds — semantic chunking, propositional chunking, agentic chunking, late chunking. Teams rewrite their ingestion pipelines, re-embed millions of documents, and then quietly discover that recall barely moved. We've shipped enough RAG systems to say it out loud: for most production workloads, fancy chunking is a distraction from the boring decisions that actually matter.

This isn't an argument against ever using semantic splitters. It's an argument for understanding where the real failure modes live, so you stop paying an embedding bill to solve a metadata problem.

The chunking strategies people actually consider

Before we get into what works, here's the landscape as of 2026:

• Fixed-size splitting — N tokens per chunk, usually with overlap. The default in LangChain, LlamaIndex, and most tutorials.
• Recursive character splitting — split on paragraphs, then sentences, then characters, until chunks fit a size budget.
• Semantic chunking — embed sentences, measure cosine distance between neighbours, split where the distance spikes. Popularised by Greg Kamradt's notebook and now baked into most frameworks.
• Propositional chunking — use an LLM to rewrite the document into atomic claims, then embed those.
• Late chunking — embed the full document with a long-context encoder, then pool token embeddings into chunk vectors (introduced by Jina AI).
• Agentic / hierarchical chunking — an LLM decides boundaries, summaries, and parent-child relationships.

The cost gradient is roughly left-to-right. The marketing claims suggest quality moves the same direction. Our evals rarely agree.

Why semantic chunking underdelivers

Semantic chunking promises natural topical boundaries. In practice it has three problems.

First, the signal is noisy. Cosine distance between adjacent sentence embeddings is a crude proxy for "topic change," especially inside technical documents where terminology repeats across sections. You end up with a threshold that works on your demo corpus and breaks on anything with tables, code, or lists.

Second, it optimises for the wrong thing. Retrieval quality depends on whether the chunk contains the answer and enough surrounding context for the LLM to use it. Semantic boundaries often cut right before the sentence that explains what a pronoun refers to. Fixed chunks with overlap accidentally solve this.

Third, it's not free. Semantic chunking requires embedding every sentence at ingestion time, often with a different model than your retrieval embedder, and then storing more metadata. For a 10M-document corpus, that's real money for a quality bump that frequently sits inside the noise floor of your eval set.

If your eval harness can't reliably tell semantic chunking from recursive splitting apart, you don't have a chunking problem — you have an evals problem or a retrieval problem.

What actually moves recall in production

Across the RAG systems we've built — legal search, internal knowledge bases, support deflection, developer docs — four things consistently matter more than the chunking algorithm.

1. Respecting structural boundaries

Documents have boundaries their authors already drew: headings, sections, list items, table rows, code blocks. Honour them. A recursive splitter configured to prefer H2 and H3 breaks will outperform semantic chunking on almost any technical corpus, because the author did the semantic work for you.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=120,
    separators=[
        "\n## ", "\n### ", "\n#### ",  # markdown headings first
        "\n\n",                          # paragraphs
        "\n",                            # lines
        ". ", " ", ""                    # fallback
    ],
    keep_separator=True,
)

For HTML or PDF, parse to a structured intermediate (markdown, or a tree) before splitting. Unstructured, Docling, and LlamaParse all do this reasonably well as of their current releases — check their docs for format support.

2. Metadata that the retriever can actually filter on

This is the single highest-leverage change we make on inherited RAG systems. A chunk without metadata is a chunk that can only be found by vector similarity. Add:

• source_id, source_type, url
• section_path (e.g. Billing > Refunds > EU customers)
• updated_at and effective_date if time matters
• product, region, audience — whatever your users actually filter by
• heading and parent_heading as separate fields

Then expose these as filters in your retriever. Half the queries that "fail" in a naive RAG system are queries where the right document was retrievable but got outranked by a stale or irrelevant one. Metadata filtering fixes that cheaply.

3. Overlap and context stitching

Fixed-size chunks lose context at boundaries. Three cheap mitigations, in order of effort:

• Token overlap of 10–20% is a free lunch in most cases.
• Prepend the heading path to every chunk. A chunk titled Refunds > EU customers > 14-day window embeds and retrieves far better than the same paragraph alone.
• Parent-document retrieval: index small chunks for precision, but return the parent section to the LLM for context. LlamaIndex and LangChain both ship this pattern.

4. Query-side work beats document-side work

If you've got a budget to spend, spend it on the query path, not re-chunking. Query rewriting, HyDE, multi-query expansion, and a decent reranker will each do more for recall@k than switching chunking strategies. We've written about rerankers separately on the blog — the short version is that a cross-encoder over the top 50 fixes a surprising amount of what people blame on chunking.

When semantic (or fancier) chunking does earn its keep

There are cases where we reach for it:

• Transcripts and chat logs without reliable structural markers. Here semantic boundaries beat arbitrary token windows because there's nothing else to anchor on.
• Long-form narrative (books, long essays) where sections aren't labelled and topics drift gradually.
• Mixed-language or OCR'd documents where structure was destroyed in preprocessing. Rebuilding boundaries from embeddings can beat broken heading detection.
• Propositional chunking for FAQ-style retrieval where users ask atomic questions and you want atomic answers. The LLM rewrite cost is high but the precision gain is real.

For everything else — product docs, codebases, policy manuals, support KBs, legal corpora with section numbers — structural splitting wins on cost and quality.

A pragmatic default stack

If you're starting a new RAG system this quarter, this is roughly what we'd ship on day one:

Ingestion
  → parse to markdown (Docling / Unstructured / LlamaParse)
  → recursive split on headings, 600–1000 tokens, 15% overlap
  → prepend heading path to each chunk
  → attach rich metadata (source, section, dates, audience)
  → embed with a current-gen model (e.g. OpenAI text-embedding-3-large,
    Voyage, or Cohere embed-v3 — check vendor docs for dimensions and pricing)

Retrieval
  → hybrid search (BM25 + vector) with metadata filters
  → top 50 → cross-encoder rerank → top 5–10
  → optional: expand to parent section before sending to the LLM

Generation
  → model chosen per task (Claude for long synthesis, GPT or Gemini
    for structured extraction — your evals decide)
  → cite chunk IDs in the response for traceability

Then — and this is the part most teams skip — build an eval set of 50-200 real queries with graded answers before you tune anything. Without it, every chunking debate is vibes.

How to actually compare chunking strategies

If you want to run the experiment on your own corpus, keep it honest:

1. Freeze your eval set and your retriever config.
2. Re-ingest the same corpus under each chunking strategy, keeping embedding model and chunk size budget constant.
3. Measure recall@k, MRR, and end-to-end answer quality (LLM-as-judge is fine if calibrated against a few hundred human labels).
4. Track cost: ingestion compute, storage, and query-time latency.
5. Report confidence intervals, not point estimates. On a 100-query eval, a 2-point recall difference is noise.

In our experience, the gap between recursive splitting and semantic chunking on structured corpora sits in the low single digits of recall@10, often inside the confidence interval. The gap between "no metadata filters" and "good metadata filters" is usually 10x bigger.

Where we'd start

If you've already got a RAG system and you're thinking about rewriting the chunker: don't, yet. Spend a week on three things first. Audit your metadata — are you filtering on what users actually care about? Add a reranker if you don't have one. Build or expand your eval set so you can measure the next change honestly. If after that your recall is still bad at the boundaries of sections, then try semantic or propositional chunking on the subset of documents where structure is genuinely missing. You'll spend less, learn more, and ship a system that holds up when the corpus doubles.

If you want a second pair of eyes on an existing RAG pipeline, that's the kind of work our team does — see our AI engineering services for how we usually scope it.