Chunking and Reranking in RAG: The Two Dials That Matter More Than Your Model

After a decade and change of building data and ML systems, I can tell you where most RAG projects go wrong. It is almost never the model. Teams ship a pipeline, get mediocre answers, and reach for a bigger LLM. The fix they actually need is upstream: chunking and reranking in RAG are the two dials that decide whether the model ever sees the right text. Retrieval controls recall. Reranking controls precision. The generator just writes up whatever those two stages hand it.

This post walks through both dials the way I tune them in production: what chunking really does, which strategies earn their keep, why vector search alone ranks things wrong, and how a cross-encoder reranker fixes it. Working Python included.

Why chunking decides your RAG ceiling

Chunking is the step where you split documents into pieces before embedding them. It sounds like plumbing. It is actually the ceiling on everything downstream, because a chunk is the smallest unit your system can ever retrieve. If the answer to a question is smeared across two chunks, or buried in a chunk about three other topics, no reranker and no LLM will recover it cleanly.

Take a 100-page employee handbook. Embed it as one vector and that vector has to represent leave policy, insurance, remote work, and ninety other topics at once. Ask “how many annual leave days do employees get?” and the math has to match your one-line question against a soup of everything. Split the handbook into sections and the leave-policy chunk can match on its own terms.

Here is the full flow, so we agree on terms:

documents -> chunk -> embed -> vector DB
query -> embed -> retrieve top-K candidates -> rerank -> top-N to the LLM -> answer

Two numbers matter in that flow. Top-K (how many candidates retrieval pulls) sets your recall budget. Top-N (how many survive reranking) sets what the LLM actually reads. Everything in this post is about making those two cuts well.

Chunk size and overlap: pick boring numbers, then measure

Chunk size is measured in tokens, not words, because your embedding model and your LLM both bill and truncate in tokens. Most teams I have worked with land somewhere between 400 and 700 tokens per chunk, with 10 to 20 percent overlap, and that range is a fine starting point. Small chunks (100 to 200 tokens) retrieve precisely but lose surrounding context. Large chunks (800 tokens and up) preserve context but drag irrelevant text into the prompt and blur the embedding across multiple ideas.

Use a real tokenizer for this, not len(text) / 4:

import tiktoken

def chunk_by_tokens(text: str, chunk_size: int = 500, overlap: int = 75):
    enc = tiktoken.get_encoding("cl100k_base")
    tokens = enc.encode(text)
    chunks, step = [], chunk_size - overlap
    for start in range(0, len(tokens), step):
        window = tokens[start:start + chunk_size]
        chunks.append(enc.decode(window))
        if start + chunk_size >= len(tokens):
            break
    return chunks

A word on overlap, because it gets treated as gospel. Overlap exists to rescue references that straddle a boundary: “These points expire after twelve months” is useless if “these points” lives in the previous chunk. That said, recent systematic write-ups, including Firecrawl’s 2026 chunking guide, report cases where overlap added indexing cost and no measurable retrieval benefit. My advice: start at 50 to 75 tokens, then test against your own retrieval metrics. Overlap is a tunable, not a religion. Too much of it gives you duplicate hits, a fatter index, and repeated text in the prompt.

Chunking strategies that earn their keep

Recursive splitting: the sane default

Fixed-size splitting every N tokens is fast but happily cuts a sentence in half. Recursive splitting fixes most of that by trying natural boundaries in order: sections, then paragraphs, then sentences, only falling back to raw tokens when it must.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",
    chunk_size=500,
    chunk_overlap=75,
    separators=["\n## ", "\n\n", "\n", ". ", " "],
)
chunks = splitter.split_text(document)

Structure-aware splitting: use the document’s own joints

When the document has structure, use it. Markdown headings, HTML sections, legal clauses, FAQ pairs. Code should be chunked by function or class, never by token count; half a function embeds as noise. A config-and-glue approach here beats any clever algorithm, because the author of the document already did the semantic segmentation for you.

Semantic chunking: probably skip it

Semantic chunking splits wherever an embedding model detects a topic shift. It sounds like the obvious upgrade, and vendors sell it that way. The evidence is unkind. A NAACL 2025 study from Vectara and UW-Madison, “Is Semantic Chunking Worth the Computational Cost?”, tested it against plain fixed-size chunking across document retrieval, evidence retrieval, and answer generation, and concluded the extra compute “is not justified by consistent performance gains.” Boring splitting plus a good reranker beats clever splitting without one, and costs less. I have watched teams burn a sprint on semantic chunking that a $0 config change would have outperformed.

Parent-child: small chunks find, big chunks answer

The one advanced strategy I recommend without hesitation. Embed small child chunks (say 300 tokens) for precise retrieval, but hand the LLM the larger parent section (say 1,500 tokens) the child came from. You get pinpoint matching and enough context to answer, without embedding the big blocks at all. LangChain ships this as ParentDocumentRetriever; the idea fits in ten lines regardless of framework:

parents = chunk_by_tokens(document, chunk_size=1500, overlap=0)

index = []
for pid, parent in enumerate(parents):
    for child in chunk_by_tokens(parent, chunk_size=300, overlap=40):
        index.append({"embed_text": child, "parent_id": pid})

# Embed only embed_text. At query time, dedupe parent_ids from the
# top child hits and send parents[pid] to the LLM.

Reranking in RAG: why vector search alone ranks wrong

Vector search uses a bi-encoder: the query and each document are embedded separately, ahead of time, and compared with cosine similarity. That separation is what makes it fast enough to search millions of chunks in milliseconds. It is also why the ranking is often subtly wrong. An embedding compresses a chunk into one point in space. Negation, exact conditions, numbers, and near-identical business terms all blur together.

ZeroEntropy has a clean example: for the query “companies that did not go bankrupt,” their embedding model scored a bankruptcy filing and a profitable-company report identically at 0.57. The reranker, a cross-encoder, separated them cleanly. Same failure shows up in every domain. Ask “can expired points be restored?” and pure similarity happily ranks “points expire after twelve months” first, because it matches the words, while the chunk that actually answers (“an administrator can restore expired points within 30 days”) sits in fourth place.

A cross-encoder reads the query and the chunk together in one forward pass, so every query token attends to every chunk token. It outputs a direct relevance score: does this text answer this question? That is a different and better question than “are these two embeddings nearby?” The cost is that you cannot precompute anything, so you only run it on the handful of candidates retrieval already found. In ZeroEntropy’s published numbers, reranking roughly 100 candidates adds tens of milliseconds and lifted NDCG@10 by 5 to 20 percent across verticals, with the biggest gains in legal, health, and finance. Elastic documents the same two-stage pattern as the standard way to ship search now.

The minimal working version, with an open-source model:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

query = "Can expired points be restored?"
candidates = [
    "Points expire twelve months after the last transaction.",
    "Customers can redeem active points for rewards.",
    "An administrator can restore expired points within 30 days.",
]
scores = reranker.predict([(query, c) for c in candidates])
for s, c in sorted(zip(scores, candidates), reverse=True):
    print(f"{s:.3f}  {c}")

Wiring the two dials together

Here is the shape of a production pipeline, end to end:

def answer(query: str, top_k: int = 30, keep: int = 5) -> str:
    q_vec = embed(query)                        # bi-encoder
    candidates = vector_db.search(q_vec, top_k) # recall stage
    scores = reranker.predict([(query, c.text) for c in candidates])
    best = [c for _, c in sorted(zip(scores, candidates),
            key=lambda t: t[0], reverse=True)[:keep]]
    context = "\n\n".join(c.text for c in best)
    return llm(f"Answer using only this context.\n\n{context}\n\nQ: {query}")

My usual starting config: 500-token chunks, 75-token overlap, retrieve top 30, rerank all 30, keep 5. Then measure and move one dial at a time.

The senior-engineer part is remembering that these dials are coupled. Shrink your chunks and you need a higher top-K to reassemble context. Raise overlap and your reranker starts seeing near-duplicates, so your top 5 collapses into two distinct facts. And the failure mode that bites hardest: a reranker can only reorder what retrieval found. If the right chunk never made the top 30, no reranker on earth will surface it. When answers are wrong, check recall first (is the right chunk in the candidate set at all?), and only then tune precision.

If you are on a managed stack such as Amazon Bedrock AgentCore rather than a hand-rolled pipeline, the same two dials exist, just behind different knobs. The vocabulary transfers; so do the failure modes. And if you want to go one level deeper into why embeddings miss what they miss, our piece on how researchers read a model’s internal workspace is a good companion read.

FAQ: chunking and reranking in RAG

What chunk size should I start with?

400 to 700 tokens with 50 to 75 tokens of overlap, split recursively at natural boundaries. Then evaluate on your own queries. Factoid lookups tolerate smaller chunks; multi-step explanations want bigger ones or a parent-child setup.

Do I always need a reranker?

No. If your corpus is small and queries are simple keyword-ish lookups, vector search alone may rank fine. Add a reranker when you see the right chunk retrieved but ranked below wrong ones. That symptom is exactly what cross-encoders fix.

How many candidates should I rerank?

Retrieve 20 to 50, rerank all of them, keep 3 to 8. Reranking is per-pair compute, so never point it at the whole corpus; that is what the vector index is for.

Retrieval score vs reranking score: what is the difference?

The retrieval score is geometric similarity between two independently computed embeddings. The reranking score is a judgment of how directly the chunk answers the exact question, computed by reading both together. Trust the second for final ordering.

If your RAG answers are disappointing, resist the model upgrade for one sprint. Log your retrieved candidates, look at what the LLM was actually given, and tune the two dials. It is unglamorous work, and it is where the quality lives. For more hands-on pipeline write-ups like this, subscribe to the blog or ping me with the failure mode you are staring at.