RAG · Module 8 of 14 · 6 concepts · about 8 min

Reranking

A second, slower and more accurate model that re-sorts the retrieved candidates so that the best few reach the prompt. One of the highest-value additions to a basic RAG system.

At work: The precision step in search products: making sure the three passages the model reads, or the three results a user sees, are the best of the fifty that were found.

Two-stage retrieval: retrieve wide, rerank narrow

What it is

Search is done in two stages.

Stage one (the retriever) is fast and approximate: it pulls 50 to 100 candidate chunks out of millions.
Stage two (the reranker) is slow and accurate: it examines each of those candidates closely against the question and puts them in a better order.

Only the top few after reranking go into the prompt.

Why it matters

Neither model could do the job alone.

The accurate model is far too slow to run over millions of chunks.
The fast model is not precise enough to be trusted with the final order.
Together, the fast one makes sure the right chunk is somewhere in the list, and the accurate one makes sure it is at the top.

In the real world: A job site retrieves 200 roughly matching vacancies in 20 milliseconds, then spends 150 milliseconds reranking them so the ten shown first are the best fits.

Visual example

Two stages

Stage one protects recall (do not miss it). Stage two protects precision (put it first).

What each stage is responsible for

	stage 1: retriever	stage 2: reranker
Looks at	Millions of chunks	50 to 100 candidates
Model	Bi-encoder and/or BM25	Cross-encoder
Speed per item	Microseconds (precomputed)	Milliseconds (computed live)
Accuracy	Good	Better
Job	Get the right chunk into the list	Move it to the top
Output	Top 50 to 100	Top 3 to 10 for the prompt

Expecting a reranker to fix a first stage that missed. It can only reorder what it is given. If the right chunk is not among the candidates, reranking changes nothing; widen or improve the first stage instead.

Cross-encoder reranking

What it is

A cross-encoder takes the question and one chunk together as a single input and outputs one number: how relevant this chunk is to this question. Compare the bi-encoder from the Embeddings module, which embeds the two separately and never lets them meet inside the model.

Why it matters

Reading both together lets the model compare them word by word: it can see that the question asks about late orders while the chunk is about damaged ones, or that the chunk says 'not eligible'.

That is why it is more accurate.
It is also why it is slow: nothing can be computed ahead of time, and it needs one full model run per candidate chunk, for every question.

In the real world: A legal search tool's reranker pushes the clause that actually answers the question above three others that merely share its vocabulary.

Visual example

Comparison

The same trade appears everywhere in search: precompute and be fast, or read together and be accurate.

Bi-encoder versus cross-encoder

	bi-encoder (retriever)	cross-encoder (reranker)
Input	Question alone; chunk alone	Question and chunk together
Output	A vector for each	One relevance score for the pair
Chunks precomputed?	Yes, at indexing time	No, impossible
Work per question	1 embedding + a vector search	1 model run per candidate
Sees fine detail (negation, conditions)	Poorly	Well
Used for	Searching millions	Re-sorting 50 to 100

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

pairs = [(question, chunk.text) for chunk in candidates]      # 50 pairs
scores = reranker.predict(pairs)                             # 50 relevance scores

ranked = sorted(zip(candidates, scores), key=lambda p: p[1], reverse=True)
top_chunks = [chunk for chunk, _ in ranked[:5]]

Using a cross-encoder as the first-stage retriever. Scoring a million chunks one pair at a time would take minutes per question. It is strictly a second-stage tool.

Practise it: Design a Semantic Search System, Design a Production RAG System

How many to retrieve and how many to keep

What it is

Two numbers to choose.

Candidates in: how many chunks the first stage hands to the reranker, usually 50 to 100.
Chunks out: how many survive into the prompt, usually 3 to 10.

A relevance cut-off on the reranker's score can also drop weak chunks, so that fewer than the maximum are kept when few are relevant.

Why it matters

More candidates in means a better chance that the right chunk is among them, at a reranking cost that grows in step.

Past a point there is nothing more to find.
Measure it: plot how often the right chunk is in the top 20, 50, 100 and 200 from the first stage, and stop where the curve flattens.
Reranker scores are also far better than embedding similarities for deciding 'is anything here relevant at all?'.

In the real world: A support team finds the right article in the first-stage top 50 for 96 percent of questions and in the top 200 for 97 percent, so it reranks 50 and saves the extra time.

Always keeping exactly k chunks. If only two candidates are relevant, sending five means three are noise the model must ignore. Keep up to k, and drop those below a score cut-off.

The latency cost of reranking

What it is

Reranking time grows with the number of candidates and with the length of each chunk, since every pair is read in full. On a GPU, reranking 50 short chunks takes on the order of tens to a couple of hundred milliseconds; on a CPU, or with long chunks, it can be several times that.

Why it matters

It is usually the slowest step before the language model itself, so it has to fit the response-time budget. The levers: fewer candidates, shorter chunks, a smaller reranker model, running on a GPU and scoring all pairs in one batch, or a hosted reranking API.

In the real world: A chat product cuts its reranking step from 600 to 120 milliseconds by moving it to a GPU and scoring all 50 candidates in one batch instead of one at a time.

Reranking long parent chunks. With parent-child retrieval, rerank the small child chunks and swap in the parents afterwards; otherwise each pair is thousands of tokens and the step becomes the bottleneck.

Practise it: Design a Semantic Search System, Design a Production RAG System

LLM rerankers

What it is

A general language model can also rerank. Give it the question and the candidate chunks, and ask it either to score each chunk for relevance or to return the chunks in order of usefulness.

Why it matters

It needs no specialised model and can follow instructions a cross-encoder cannot, such as 'prefer the most recent policy' or 'prefer official documents over forum posts'.

The costs are higher price and delay, and scores that are less consistent from run to run.
It suits low-volume, high-value questions more than a busy consumer chat.

In the real world: A due-diligence tool, answering a few hundred expensive questions a day, uses a language model to rerank with the instruction to prefer signed contracts over drafts.

Asking the model to order thirty long chunks in a single prompt. Models lose track in long lists and may drop or repeat items. Score chunks individually or in small groups, and check that every ID you sent comes back exactly once.

Late interaction (ColBERT)

What it is

A middle path between the two encoder types.

A late-interaction model such as ColBERT keeps one vector per token of the chunk, not one vector for the whole chunk.
At query time, each query token is matched to its most similar chunk token, and those best matches are added up into a score.

Why it matters

Chunk token vectors can still be computed ahead of time, like a bi-encoder, but the matching happens at the level of individual words, which recovers much of a cross-encoder's precision. The price is storage: many vectors per chunk instead of one, often ten or more times the index size.

In the real world: A patent-search company uses late interaction because matching on individual technical terms matters, and accepts an index many times larger to get it.

Adopting it for its benchmark scores without pricing the index. Storing a vector per token across a hundred million chunks is a very different infrastructure bill from storing one per chunk.