Agentic AI · Module 11 of 41 · 10 concepts · about 16 min

Agentic RAG

How retrieval changes when an agent controls it: rewriting queries, retrieving in several steps, choosing between sources, checking what came back, and the corrective and self-reflective variants; with a clear comparison against the fixed RAG pipeline.

At work: What a team builds when its RAG chatbot answers simple questions well and fails the ones that need two look-ups, a comparison or a different data source.

Traditional RAG and where it stops

What it is

RAG (retrieval-augmented generation) answers from your documents in a fixed pipeline:

Retrieve: search a document index for passages similar to the question.
Augment: put the top passages into the prompt.
Generate: the model answers from those passages.

One search, one answer, every time, in an order your code fixed.

Why it matters

That pipeline is cheap and predictable, and it fails on four kinds of question:

Multi-hop: 'Is the fee for my late order refundable?' needs the order first, then the policy.
Vague or conversational: 'what about the other one?' cannot be searched as written.
Wrong source: the answer is in the orders database, not the help centre.
Bad retrieval: the search returned nothing relevant, and the pipeline answers anyway.

Each failure is the pipeline having no way to look at what it got and decide what to do next.

In the real world: A help-centre bot answers 'what is the return window?' perfectly and fails 'can I still return the kettle I bought?', which needs the purchase date from another system.

Blaming the model for a retrieval failure. If the right passage was never fetched, no model can answer correctly. Check what was retrieved before changing anything else.

RAG pipeline versus agentic RAG

What it is

Agentic RAG makes retrieval a tool the model controls, instead of a fixed first step. The model decides:

whether to retrieve at all;
what to search for, and how to phrase it;
where to search: which index, database or API;
whether the results are good enough, or another search is needed;
when it has enough to answer.

Why it matters

It trades predictability for capability.

It handles multi-step and multi-source questions that a pipeline cannot.
It costs several model calls instead of one, takes longer, and can loop.
Use the pipeline for simple factual questions and the agent for the rest; a router can send each question to the right one.

In the real world: A company routes each question by complexity: about four in five go through the plain pipeline in under two seconds, and the remainder go to an agentic path that takes eight.

Visual example

Comparison

The pipeline is a recipe. The agent is a researcher.

RAG pipeline versus agentic RAG

	RAG pipeline	agentic RAG
Who decides to search	Your code, always	The model, when needed
Searches per question	One	Zero to several
Sources	One index	Several, chosen per question
Checks its results	No	Yes
Multi-hop questions	Fails	Handles
Latency	One to two seconds	Several seconds
Cost	One model call	Several
Failure mode	Answers from bad passages	Loops or over-searches

Replacing a working RAG pipeline with an agent for every question. Most questions in a help centre are single-hop. Paying for a loop on all of them multiplies cost and delay for no gain.

Query rewriting

What it is

Query rewriting turns the user's words into a better search. Common forms:

Stand-alone rewriting: replace 'it' and 'the other one' using the conversation, so the query makes sense alone.
Decomposition: split a compound question into separate searches.
Expansion: add synonyms or the terms documents actually use ('money back' becomes 'refund').
Hypothetical answer: write a plausible answer and search for passages similar to it. Known as HyDE.
Step-back: search for the general rule behind a specific case.

Why it matters

People do not phrase questions the way documents are written.

A search for 'what about the other one?' retrieves nothing useful.
Search quality depends more on the query than on almost any other setting.
In an agent, rewriting is free: the model writes the search query itself as the tool's input.

In the real world: A support search's hit rate on follow-up questions rises from about half to nine in ten when a rewriting step resolves pronouns before retrieval.

REWRITE_PROMPT = """Rewrite the user's latest message as search queries for a help centre.
- Make each query stand alone: resolve 'it', 'that', 'the other one' from the conversation.
- If the message asks about two things, write two queries.
- Use the words a policy document would use.
Return JSON: {{"queries": ["...", "..."]}}

Conversation:
{history}
Latest message: {message}"""

# "ok and what about the damaged one?"  ->
# {"queries": ["refund policy for a damaged item", "time limit to report a damaged item"]}

Rewriting so freely that the query drifts from what was asked. Keep the user's key terms, especially names, ids and numbers, in the rewritten query.

Multi-step and iterative retrieval

What it is

Two ways an agent retrieves more than once:

Multi-step (multi-hop) retrieval: each search uses the result of the last. First find which plan the customer is on, then search that plan's terms.
Iterative retrieval: search, read, notice what is still missing, search again with a sharper query, until the evidence is sufficient or a limit is reached.

Why it matters

Some answers do not exist in any single passage.

A comparison needs both sides fetched.
A chain of facts needs each link fetched in turn.
Iteration lets a weak first search be repaired instead of being answered from.
A cap on rounds is essential: without one, 'search again' becomes a loop.

In the real world: A legal assistant answering 'does clause 9 override the master agreement?' fetches clause 9, sees it refers to schedule B, fetches schedule B, and only then answers.

Visual example

Multi-hop

The second search could not have been written before the first returned.

Answering a question that needs two look-ups

Question“Is the fee on my late order refundable?”
Hop 1get_order → 62 minutes late
Hop 2Policy search → refund if > 30 min
Enough evidence?Lateness and rule both known
AnswerYes, with both sources cited

def iterative_retrieve(question: str, max_rounds: int = 3) -> list[str]:
    evidence, query = [], question
    for _ in range(max_rounds):
        evidence += search(query, k=4)
        verdict = llm_json(
            "Question: {q}\nEvidence so far:\n{e}\n\n"
            'Return {{"sufficient": true|false, "missing": "...", "next_query": "..."}}'
            .format(q=question, e="\n".join(evidence))
        )
        if verdict["sufficient"]:
            break
        query = verdict["next_query"]            # a sharper search for what is missing
    return dedupe(evidence)

Accumulating every retrieved passage across rounds. By round three the context is mostly near-duplicates. Deduplicate, and keep only passages that were judged relevant.

Practise it: Design a Tool-Calling Agent Over 500 APIs

Retrieval planning and source selection

What it is

Two decisions the agent makes before searching:

Source selection (retrieval routing): which store holds the answer. Help-centre articles, the orders database, product catalogue, the live web. Each is a separate tool with a description of what it contains.
Retrieval planning: for a complex question, list the look-ups needed and their order before starting, as in module 7.

Why it matters

Real knowledge lives in several places with different shapes.

Policy text suits similarity search; an order's status needs an exact database query; today's prices need an API.
Searching the wrong source returns confident, irrelevant text.
A short retrieval plan avoids both missing a needed look-up and making redundant ones.

In the real world: An enterprise assistant chooses between the HR handbook, the IT wiki and the ticket system per question; before routing, HR questions regularly came back with IT answers.

RETRIEVAL_TOOLS = [
    {"name": "search_help_centre",
     "description": "Policies and how-to articles. Use for rules: refunds, fees, "
                    "delivery areas. Not for a specific customer's orders."},
    {"name": "query_orders",
     "description": "This customer's orders: items, totals, delivery times, status. "
                    "Use for anything about a particular order."},
    {"name": "search_catalogue",
     "description": "Products, prices and stock right now. Use for availability or price."},
]
# The model reads the three descriptions and picks. Clear 'use for' and
# 'not for' lines are what make the choice reliable.

One giant index containing policies, orders, tickets and product data together. Similarity search then mixes them, and access rules become impossible. Separate sources, separate tools, separate permissions.

Practise it: Design a Multi-Agent AI System, Design a Tool-Calling Agent Over 500 APIs

Retrieval validation

What it is

Retrieval validation (or grading) is checking retrieved passages before they are used. Each passage is judged:

Relevant? Does it bear on the question at all?
Sufficient? Taken together, is there enough to answer?
Trustworthy and current? Is the source authoritative, and is the document in date?

Irrelevant passages are dropped. If nothing useful remains, the agent searches again or says it does not know.

Why it matters

A model given irrelevant text tends to use it anyway.

Dropping bad passages improves answers more than adding good ones.
A cheap, small model can do the grading.
'I could not find this' is a better answer than a fluent one built on the wrong document.

In the real world: A medical information assistant drops any passage older than its review date; an out-of-date dosage page stops appearing in answers the day the check is added.

def grade(question: str, passages: list[str]) -> list[str]:
    """Keep only passages a small model judges relevant."""
    kept = []
    for passage in passages:
        verdict = llm_small_json(
            f"Question: {question}\nPassage: {passage}\n"
            'Does the passage help answer the question? Return {"relevant": true|false}'
        )
        if verdict["relevant"]:
            kept.append(passage)
    return kept

Grading relevance but never sufficiency. Three relevant passages that each cover a different half of the question still do not answer it. Ask 'is this enough?' as its own check.

Practise it: Design a Tool-Calling Agent Over 500 APIs

Corrective RAG

What it is

Corrective RAG (CRAG) adds a decision after retrieval. The retrieved passages are graded, and one of three paths is taken:

Correct: the passages are good. Trim them to the relevant sentences and answer.
Incorrect: the passages are poor. Discard them and fall back to another source, typically a web search or a different index.
Ambiguous: mixed. Use the good parts and add the fallback as well.

Why it matters

It gives the system a response to bad retrieval other than answering anyway.

The grade is made by a lightweight evaluator, so the extra cost is small.
The fallback makes the system robust to gaps in the index.
It is a fixed, bounded flow, so it is easier to test than a free agent loop.

In the real world: A product-support assistant grades its manual search as poor for a newly launched model, falls back to the release notes index, and answers correctly instead of describing last year's model.

Visual example

Corrective RAG

The grader is what a plain pipeline lacks.

Retrieve, grade, then choose a path

RetrieveTop passages for the question
GradeCorrect, ambiguous or incorrect
either
If correctKeep the relevant sentences
If incorrectRewrite; search a fallback source
GenerateFrom surviving evidence, cited

Falling back to open web search for questions about your own private policies. The web will return a competitor's policy. The fallback source must be one that can actually contain the answer.

Self-RAG

What it is

Self-RAG (self-reflective RAG) has the model check itself at three points:

Before: does this question need retrieval at all? 'Thanks!' does not.
After retrieving: is each passage relevant?
After answering: is every statement in the answer supported by the passages, and does the answer actually address the question?

If the answer is unsupported or off the point, it is regenerated or more is retrieved.

Why it matters

It attacks the two failures users notice most.

Unnecessary retrieval wastes time and can pull in distracting text.
An answer that goes beyond its sources is a hallucination with citations attached, which is worse than one without.
The final 'is it supported?' check is the most valuable single step in any RAG system, agentic or not.

In the real world: A bank's assistant refuses to send any answer whose support check fails; its rate of answers contradicted by the source document falls to near zero, at the price of more 'I could not find this' replies.

Visual example

Variants

Each adds decisions the one before lacks.

Plain, corrective, self-reflective and agentic RAG

variant	decides to retrieve	grades passages	has a fallback	checks the answer	multi-step
Plain RAG	No	No	No	No	No
Corrective RAG	No	Yes	Yes	No	No
Self-RAG	Yes	Yes	Retrieves again	Yes	Limited
Agentic RAG	Yes	Yes	Any tool	Yes	Yes

def answer_with_self_check(question: str, max_attempts: int = 2) -> str:
    if not needs_retrieval(question):                       # check 1
        return llm(question)
    passages = grade(question, search(question))            # check 2
    for _ in range(max_attempts):
        draft = generate(question, passages)
        check = llm_json(SUPPORT_PROMPT.format(answer=draft, passages=passages, q=question))
        if check["fully_supported"] and check["answers_question"]:     # check 3
            return draft
        passages += grade(question, search(check["missing"]))
    return "I couldn't find enough in our documents to answer that reliably."

Running all three checks with the largest model on every question. The checks are simple judgements; a small model does them well at a fraction of the cost.

Interview questions: agentic RAG

What it is

The questions interviewers ask about agentic RAG, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: Asked 'what is agentic RAG?', a candidate draws the pipeline, then draws a loop around the retriever with a grader in it; the diagram is the answer.

Q: What is agentic RAG?

Retrieval as a tool the model controls. It decides whether to search, what query to use, which source, whether the results are good enough, and when to stop. A plain pipeline always does one search then answers.

Q: When is plain RAG enough?

For single-hop factual questions against one corpus. It is cheaper, faster and more predictable. A router can send only complex questions to the agentic path.

Q: What is query rewriting?

Converting the user's message into effective searches: making it stand-alone, splitting compound questions, using document vocabulary, or generating a hypothetical answer to search with.

Q: What is multi-hop retrieval?

Retrieval where a later search depends on an earlier result, such as finding the customer's plan and then that plan's terms. A fixed single search cannot do it.

Q: Explain corrective RAG.

Retrieve, grade the passages, then branch: use them if good, discard and fall back to another source if bad, combine if mixed. It gives the system a response to poor retrieval.

Q: Explain self-RAG.

The model decides whether to retrieve, judges passage relevance, and checks that the answer is supported by the passages and answers the question, regenerating or retrieving more if not.

Q: How do you stop an agentic RAG system looping?

Cap retrieval rounds, detect repeated queries, require each new query to target something specific that is missing, and return 'not found' when the cap is reached.

Q: How do you evaluate it?

Retrieval quality (was the needed passage fetched), groundedness (is each claim supported), answer correctness, plus agent measures: searches per question, latency and cost.

Defining agentic RAG as 'RAG with an agent framework'. The defining feature is who decides: the model, not fixed code.

Exercise: upgrade a RAG pipeline to agentic RAG

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: A documentation team runs this comparison on its own help centre and finds that one question in six needs the agentic path, which becomes the routing rule.

Your task. Build a plain RAG pipeline over a few documents, find the questions it fails, and fix them by adding rewriting, grading and a second source.

Steps

Use the six Kirana Kart help passages from the RAG topic (or write six of your own) and a simple search: word overlap is enough.
Build the plain pipeline: search once, answer from the top two passages.
Write eight questions: four simple ones, two that need an order looked up first, and two the documents cannot answer.
Add a query_orders tool over fake data and let the model choose between it and search_help_centre.
Add a relevance grader and a 'sufficient?' check with at most three retrieval rounds.
Add a final support check: refuse to answer if a claim is not in the evidence.

You are done when

The plain pipeline fails at least three of the eight questions, and you can say why for each.
The agentic version answers the two multi-hop questions correctly.
For the two unanswerable questions it says it does not know instead of inventing.
You can report searches per question for each of the eight.

Stretch. Add a router that sends simple questions to the plain pipeline and compare average cost with sending everything to the agent.