Semantic Chunking vs Fixed-Size Chunks: What Actually Moves RAG Quality

Every RAG project hits the same fork in the road around week two: someone reads a blog post about semantic chunking, and suddenly the fixed-size splitter that was working fine feels embarrassing. Before you rip it out, it's worth being honest about what chunking actually does for retrieval quality — and what it doesn't.

This is the breakdown we wish we'd had before spending a sprint rewriting a splitter that moved recall by less than two points.

What each strategy actually does

Fixed-size chunking splits text on token or character counts, usually with some overlap. LangChain's RecursiveCharacterTextSplitter is the reference implementation most teams start with: it tries to split on paragraph, then sentence, then word boundaries, falling back to hard cuts when it has to.

Semantic chunking splits based on meaning. The common approach — popularised by Greg Kamradt and now available in LangChain, LlamaIndex, and various standalone libraries — embeds each sentence, then cuts wherever the cosine distance between consecutive sentences exceeds some threshold. The idea: a chunk should be about one thing.

There are hybrids too. Docling and Unstructured produce structure-aware chunks that respect headings, tables, and lists. Anthropic's contextual retrieval prepends a short LLM-generated summary to each chunk so the embedding carries document-level context. These are worth knowing about because they often matter more than the split algorithm itself.

The mental model that helps

Chunking is not really about splitting. It's about deciding what unit of text your embedding model has to represent as a single vector, and what unit your LLM has to reason over when it lands in the context window. Those two units don't have to be the same thing, which is a point we'll come back to.

Where fixed-size wins

Fixed-size chunking has three things going for it, and they matter more than most tutorials admit.

It's predictable. You know your token budget per chunk, which means you know how many chunks fit in context, which means your cost per query is stable. Semantic chunking produces variable-length chunks — sometimes a single sentence, sometimes eight paragraphs — and that variance leaks into every downstream cost calculation.

It's cheap to build. No embedding pass at ingest time just to figure out where to cut. For a corpus of a few million documents, that difference is real money.

It's good enough for prose. If your source is reasonably well-written articles, documentation, or transcripts, a 512-token chunk with 15% overlap will get you most of the way there. The sentences inside a paragraph are usually about the same thing already — the author did the semantic chunking for you.

In our experience, the teams that get the best results from fixed-size chunking are the ones who tune two things: chunk size (try 256, 512, 1024 and actually measure) and overlap (10 – 20% is the useful range; more than that mostly wastes storage).

Where semantic chunking earns its keep

Semantic chunking pays off when your source documents violate the assumption that adjacent sentences belong together. That's more common than you think:

• Mixed-topic pages. Wiki-style internal docs where one page covers deployment, billing, and onboarding under different H2s.
• Meeting transcripts. Speakers jump topics without paragraph breaks.
• Legal and policy documents. Long clauses with embedded exceptions that a fixed splitter will guillotine mid-thought.
• Chat logs and support tickets. Turn boundaries matter, but a fixed splitter doesn't see them.

On these, semantic chunking meaningfully improves retrieval precision because you stop returning chunks that are half about the thing the user asked and half about something else — which is exactly the kind of noise that makes an LLM hallucinate confidently.

A minimal semantic splitter

You don't need a library to try this. The core loop is small:

import numpy as np
from openai import OpenAI

client = OpenAI()

def embed(texts):
    resp = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts,
    )
    return np.array([d.embedding for d in resp.data])

def semantic_chunks(sentences, threshold_percentile=90, min_chunk_sents=2):
    embeddings = embed(sentences)
    # cosine distance between consecutive sentences
    dists = [
        1 - np.dot(embeddings[i], embeddings[i+1])
        for i in range(len(embeddings) - 1)
    ]
    cut_threshold = np.percentile(dists, threshold_percentile)

    chunks, current = [], [sentences[0]]
    for i, d in enumerate(dists):
        if d > cut_threshold and len(current) >= min_chunk_sents:
            chunks.append(" ".join(current))
            current = [sentences[i+1]]
        else:
            current.append(sentences[i+1])
    chunks.append(" ".join(current))
    return chunks

Two notes on this. First, the percentile approach is more robust than a fixed cosine threshold because it adapts to how varied the document is. Second, min_chunk_sents matters — without it, you get single-sentence chunks that lack the context an LLM needs to actually answer questions.

The thing that actually moves the needle

Here's the uncomfortable truth from running evals across a few dozen RAG deployments: swapping fixed-size for semantic chunking usually moves retrieval quality by a few percentage points. Sometimes less. Occasionally it makes things worse because your variable-length chunks now have embeddings that represent them less faithfully.

The changes that consistently move quality by double digits are:

1. Contextual retrieval. Prepend a one-line summary of the source document (or section) to each chunk before embedding. Anthropic published results showing large recall improvements from this, and it matches what we've seen. Cheap to implement, works with any chunking strategy.
2. Hybrid search. Combine dense vectors with BM25 or a keyword index. Semantic search misses exact identifiers, error codes, and jargon that keyword search nails.
3. Rerankers. A cross-encoder on the top 50 results reorders them by actual relevance to the query, not just embedding similarity. Cohere Rerank and Voyage rerankers are the usual choices.
4. Better queries. Query rewriting, HyDE, or multi-query retrieval often beats better chunking, because the retrieval problem is symmetric — the query side matters as much as the document side.

If you haven't done these first, arguing about chunking strategy is premature.

Decoupling the retrieval unit from the context unit

One pattern worth adopting regardless of which chunker you pick: embed small, retrieve small, but pass large to the LLM.

Store your chunks as small units (a paragraph, a semantic block) with a pointer to their parent section. Retrieve on the small units for precision. Then, when you assemble the LLM prompt, expand each hit to include its parent section or neighbouring chunks. LlamaIndex calls this the parent-document retriever pattern; LangChain has an equivalent.

This gets you the recall benefits of tight chunks without the context-starvation problem of feeding an LLM three isolated sentences and asking it to reason.

Measuring it honestly

Don't take anyone's word on this — including ours. Build a small eval set of 30 – 100 real questions from your domain, label the correct source passages, and measure recall@k and MRR under each chunking strategy. Ragas, TruLens, and Braintrust all have off-the-shelf harnesses for this.

The thing to watch for: retrieval metrics can improve while end-to-end answer quality gets worse, or vice versa. Always evaluate the full pipeline, not just the retriever, before declaring victory. Our AI engineering team has written more about eval design in other posts on the blog — the short version is that without evals, every chunking debate is vibes.

Cost and latency footprint

A rough sense of what each choice costs:

• Fixed-size ingest: one embedding call per chunk. Predictable.
• Semantic chunking ingest: one embedding call per sentence (for split decisions) plus one per final chunk. Roughly 2 – 4× the ingest embeddings, depending on sentence length.
• Contextual retrieval ingest: add one LLM call per chunk to generate the context prefix. This is the expensive one — mitigated significantly by prompt caching if your provider supports it.

At query time, chunking strategy is invisible to latency. What you paid was paid at ingest.

Where we'd start

If you're building a new RAG system in 2026, our default order of operations: ship with RecursiveCharacterTextSplitter at 512 tokens with 15% overlap, add hybrid search on day one, build a 50-question eval set in week one, then in this order try contextual retrieval, a reranker, and query rewriting. Only reach for semantic chunking after those, and only on the document types where fixed-size is obviously mangling meaning. You'll ship faster and end up with a better system than the team that spent a sprint tuning a semantic splitter before they had an eval to prove it worked.