RAG · Module 7 of 14 · 8 concepts · about 11 min

Query Understanding and Transformation

What a user types is rarely a good search query. These techniques rewrite, expand, split and route the question before any search runs.

At work: The step that makes a chat assistant feel like it understands: handling follow-ups such as 'and for members?', vague questions, and questions that need two different sources.

Why the raw question is a poor search query

What it is

Users write the way they talk. Their questions are:

short ("refund?");
dependent on earlier messages ("what about for members?");
multi-part ("what's the fee and can I change my address?");
vague, or wrapped in chatter ("hi, quick one, sorry to bother you, but...").

The retriever embeds exactly what it is given, so a poor query produces a poor search.

Why it matters

Query transformation puts a cheap language-model step in front of the retriever to turn what the user said into what should be searched. It is often the best-value improvement in a chat-style RAG system, because conversation makes raw questions especially bad queries.

In the real world: A telecom chatbot's accuracy on follow-up questions doubles after the team logs the search queries and sees that half of them were fragments like 'and prepaid?'.

Visual example

Before and after

Each row is a different technique from this module.

What the user typed versus what should be searched

user typed	problem	searched instead	technique
what about for members?	Depends on earlier messages	Is delivery free for Kart Plus members?	Rewriting
refund?	Too short, vague	refund policy; late order refund; damaged item refund	Multi-query
fee and can I change address?	Two questions in one	delivery fee / change delivery address	Decomposition
how many orders did I place?	Not a document question	(send to the orders database)	Routing

Blaming the retriever for results it could never have got right. Before tuning the search, print the exact text that was sent to it. If that text is 'what about for members?', no retriever on earth can find the right passage.

Query rewriting for follow-up questions

What it is

In a conversation, later questions lean on earlier ones.

Query rewriting (also called condensing the question) gives a language model the recent chat history and the new message, and asks it to write one standalone question that makes sense with no history at all.
That standalone question is what gets searched.

Why it matters

It is essential for any chat interface. 'What about for members?' has no searchable content; 'Is delivery free for Kart Plus members?' does. The rewrite is a small, fast model call, and it is the single change that fixes most multi-turn failures.

In the real world: An airline's assistant handles 'and if I cancel it?' correctly because the search runs on 'What is the cancellation fee for an economy ticket to Dubai?', built from the previous turn.

History:
  User:      How much is the delivery fee?
  Assistant: ₹25 for orders under ₹199, free above that.
New message:
  User:      what about for members?

Rewrite prompt:
  Rewrite the user's last message as a standalone question that
  can be understood without the conversation. Do not answer it.

Standalone question (this is what gets searched):
  What is the delivery fee for Kart Plus members?

Letting the rewriter answer the question or add facts of its own. It must only restate. Tell it so explicitly, and keep the original message for the final answer step so nothing the user said is lost.

Practise it: Design a Production RAG System

Multi-query expansion

What it is

A language model writes several different phrasings of the question, typically three to five. Each is searched separately, and the result lists are merged, usually with reciprocal rank fusion.

Why it matters

Any single phrasing may miss the passage because of wording.

Several phrasings cast a wider net: one of them is likely to sit close to how the document says it.
This raises the chance that the right chunk is retrieved at all, at the cost of more searches (which run in parallel and are cheap) and one extra model call.

In the real world: A medical-information service expands 'heart attack signs' into variants using 'myocardial infarction symptoms', so both lay and clinical documents are found.

variants = llm(
    "Write 3 different search queries for this question, one per line:\n"
    + question
).splitlines()

rankings = [vector_index.search(embed(v), k=20) for v in [question, *variants]]
candidates = reciprocal_rank_fusion(rankings)

Dropping the original question and searching only the variants. Generated phrasings can drift from what the user meant. Always include the original as one of the queries.

HyDE: search with a made-up answer

What it is

HyDE (Hypothetical Document Embeddings) asks a language model to write a short, plausible answer to the question without looking anything up, then embeds that made-up answer and searches with it instead of the question.

Why it matters

A question and its answer look different: one is short and interrogative, the other is a statement full of the relevant vocabulary.

A made-up answer, even a factually wrong one, looks like the real passage and so lands near it in the vector space.
It helps most with short or vague questions and with embedding models not trained for question-to-passage search.

In the real world: A scientific-literature search uses HyDE so that a four-word question retrieves papers written in dense technical prose that shares none of the question's phrasing.

Question:
  late order compensation?

Made-up answer (never shown to the user, may be wrong):
  If an order is delivered later than the promised time, the
  customer is eligible for a refund of the delivery charge,
  which is credited back to their account.

This made-up answer is embedded and used as the search query.
The real passage is retrieved and the real answer comes from it.

Using HyDE where the model's guess is likely to be badly wrong, such as questions about your private products or internal codes. A confidently wrong guess steers the search to the wrong part of the collection. It also adds a model call's delay to every question.

Query decomposition

What it is

A question that asks several things, or that needs facts from different places, is split into sub-questions. Each sub-question is retrieved separately, and the model answers from the combined results.

Why it matters

One embedding of a two-part question lands between the two topics and retrieves neither well.

"Compare the refund rules for late orders and damaged items" needs both passages; searching for each on its own finds both reliably.
Decomposition is also the first step towards multi-hop questions, where the second search depends on the answer to the first.

In the real world: An insurance assistant splits 'does my policy cover flood damage and how do I claim?' into a coverage lookup and a claims-process lookup, and answers both parts.

Question:
  Compare the refund for a late order with the refund for a damaged item.

Sub-questions:
  1. What is the refund for a late order?
  2. What is the refund for a damaged item?

Retrieve for each, then answer the original question from both sets.

Decomposing every question. A simple question split into three sub-questions triples the cost and pulls in unrelated passages. Let the model return the question unchanged when it is already a single question.

Step-back questions

What it is

A step-back question is a more general version of a very specific question.

For "my order of 3 items arrived 42 minutes late on a Sunday, do I get a refund?", the step-back question is "what is the refund policy for late orders?".
Both the specific and the general question are searched.

Why it matters

Documents state general rules, while users ask about their particular case.

An over-specific query can miss the rule because the rule mentions none of the details (3 items, Sunday, 42 minutes).
The general query finds the rule; the model then applies it to the specifics.

In the real world: A tax assistant turns 'can I claim my daughter's 2024 school bus fees?' into a general lookup of the rules on education deductions, then applies them to the case.

Searching only the step-back question. Sometimes the details matter (there may be a separate Sunday rule), so retrieve with both and let the reranker sort them.

Query routing

What it is

Routing decides where a question should go before anything is searched. A small classifier or a language model reads the question and picks a destination: the help-centre index, the product catalogue index, a database query for account data, a web search, or no retrieval at all for a greeting or general-knowledge question.

Why it matters

One index cannot serve every question well. 'What is your refund policy?' belongs to documents. 'Where is my order?' belongs to the orders database. 'Hi' needs nothing. Routing keeps each question away from sources that would only add noise, and saves the cost of retrieval when none is needed.

In the real world: A bank's assistant sends policy questions to its document index and balance questions to the core banking system, and never tries to answer 'what is my balance?' from a PDF.

Question                                 Route
---------------------------------------  ---------------------------
What is the refund policy?               help-centre documents
Where is my order ORD-88213?             orders database (lookup)
How many orders did I place last month?  orders database (count)
Do you sell oat milk?                    product catalogue
Thanks, bye!                             no retrieval

Having no fallback. A router is a classifier and will sometimes be wrong. When it is unsure, search the default index (or several), instead of confidently sending the question somewhere it cannot be answered.

Practise it: Design an Enterprise Knowledge Assistant

The cost of transforming queries

What it is

Every technique in this module adds a language-model call before retrieval starts. That costs money and, more importantly, time: typically a few hundred milliseconds each, on a path where the user is waiting.

Why it matters

A good design uses these selectively:

Rewrite only when there is conversation history.
Use a small, fast model for rewriting and routing.
Run independent steps at the same time.
Skip expansion for questions that already look like good queries.
Add a technique only after measuring that it improves retrieval on your test questions.

In the real world: A voice assistant, where every 100 milliseconds is felt, keeps only follow-up rewriting, and uses a small model for it, after measuring that the other techniques added delay for little gain.

Visual example

Cost and benefit

Use the fewest that fix the failures you actually measured.

When each technique earns its extra model call

technique	fixes	extra cost	use when
Rewriting	Follow-ups that depend on history	1 small model call	Always, in chat interfaces
Multi-query	Wording mismatch	1 call + several searches	Recall is low on varied phrasing
HyDE	Short or vague questions	1 call	The model can guess plausibly
Decomposition	Multi-part questions	1 call + several searches	Comparisons, 'and' questions
Step-back	Over-specific questions	1 call + 1 search	Documents state general rules
Routing	Wrong source searched	1 small call	More than one source exists

Stacking rewriting, multi-query, HyDE and decomposition because each one sounded good. The delays add up to seconds, and the gains do not add up: these techniques overlap heavily in what they fix.