RAG · Module 1 of 14 · 8 concepts · about 13 min

What RAG Is and Why It Exists

The problem RAG solves, the idea in one sentence, the two pipelines every RAG system has, and how it compares with fine-tuning and with simply pasting everything into the prompt.

About RAG: A language model only knows what it was trained on, so it cannot answer questions about your own documents, and it sometimes makes things up. RAG fixes this by looking up the relevant passages first and asking the model to answer from them, like an open-book exam. You will start with that idea, then build each part in order: turning text into numbers that capture meaning, preparing and cutting up documents, storing and searching them, and writing the prompt that produces a sourced answer. After that you will learn how to measure whether it works, how to find what broke when it does not, what changes with real users, and how to answer the questions interviewers ask about all of it.

At work: The first decision on any 'chat with our documents' project: support assistants, internal knowledge search, policy and legal lookup, coding help over a private codebase.

The problem: a language model only knows its training data

What it is

A large language model (LLM) such as the one behind ChatGPT or Claude learned from a huge pile of public text collected up to a certain date.

It has never seen your company's documents, and it knows nothing that happened after that date.
Ask it about Kirana Kart's refund policy and it has two choices: admit it does not know, or produce something that sounds right. It often does the second.
A confident, made-up answer is called a hallucination.

Why it matters

Three gaps follow from this, and every RAG interview starts here.

Private knowledge: your policies, tickets and contracts were never in the training data.
Fresh knowledge: the model's knowledge stops at its training cutoff.
Proof: even when the model is right, it cannot show where the answer came from, so nobody can check it.

In the real world: A telecom company puts a general chatbot on its website; it invents a plan price that does not exist, a customer screenshots it, and support has to honour or apologise.

Customer:  My order was 40 minutes late. Do I get anything back?

Model alone:  Yes! Kirana Kart offers a 20% discount coupon
              for any late order.          <- invented, sounds real

What the help centre actually says:
              If your order arrives more than 30 minutes late,
              we refund the full delivery fee to your wallet.

Thinking a bigger or newer model fixes this. A better model hallucinates less often, but it still cannot know a policy it has never read. The missing piece is information, not intelligence.

The RAG idea: look it up first, then answer

What it is

RAG stands for Retrieval-Augmented Generation. Before the model answers, the system searches your documents for the passages most related to the question, pastes those passages into the prompt, and asks the model to answer using them. The three words:

Retrieval is the search.
Augmented means the prompt is extended with what was found.
Generation is the model writing the answer.

Why it matters

It turns a closed-book exam into an open-book exam.

The model no longer has to remember your refund policy; it only has to read the passage in front of it and answer from it.
That fixes all three gaps at once: the passage is private, it is as fresh as your documents, and it can be shown to the user as the source.

In the real world: A bank's internal assistant answers staff questions about loan rules by first pulling the current circular, so the answer changes the day the circular does, with no retraining.

Visual example

RAG

The model never answers from memory. It answers from what the search step found.

One question through a RAG system

Question“My order was 40 minutes late…”
RetrieveFind the late-order refund passage
AugmentPassage + question in one prompt
GenerateAnswer from the passage, with source

If step 2 finds the wrong passage, step 4 cannot be right. Retrieval quality sets the ceiling.

1. Question:   My order was 40 minutes late. Do I get anything back?

2. Retrieved:  [Late order refund] If your order arrives more than
               30 minutes late, we refund the full delivery fee
               to your wallet.

3. Prompt:     Answer using only the passage below.
               Passage: <the retrieved text>
               Question: <the customer's question>

4. Answer:     Yes. Because your order was more than 30 minutes
               late, the delivery fee is refunded to your wallet.
               [source: Late order refund]

Believing RAG changes the model. It does not. The model's weights are untouched; only the prompt changes. That is why RAG is cheap to update (edit a document) and also why it fails whenever the search step brings back the wrong passage.

The two pipelines: indexing and answering

What it is

Every RAG system is two separate pipelines.

The indexing pipeline runs ahead of time, in the background: load the documents, cut them into pieces, turn each piece into numbers, store them in a searchable index.
The query pipeline runs live, each time a user asks something: turn the question into numbers, search the index, build the prompt, generate the answer.

Why it matters

They have opposite needs, and interviewers expect you to keep them apart.

Indexing can be slow and thorough, because nobody is waiting.
Answering must be fast, usually a couple of seconds.
Almost every design decision belongs to exactly one of the two, and a change to the indexing side (a new chunk size, a new embedding model) means rebuilding the index, while a change to the query side takes effect immediately.

In the real world: An e-commerce company re-indexes its product manuals overnight in a batch job, while the live assistant answers shoppers in under two seconds from the index that job produced.

Visual example

Two pipelines

Indexing happens once per document, ahead of time. Answering happens once per question, live.

The two halves of a RAG system

step	indexing pipeline (offline)	query pipeline (online)
1	Load and parse documents	Receive the question
2	Clean and cut into chunks	Rewrite it into a good search query
3	Embed each chunk (text to numbers)	Embed the query with the same model
4	Store vectors, text and metadata	Search the index, then rerank
5	Keep it up to date as documents change	Build the prompt and generate the answer

Both columns meet at the index: one writes to it, the other reads from it.

Describing RAG as one straight line. If you draw a single pipeline in an interview, the first follow-up will be 'what happens when a document changes?', and you will have nowhere to put the answer. Draw two.

The vocabulary: document, chunk, embedding, index, retriever, context

What it is

Six words carry the whole subject.

A document is one source file: a help article, a PDF.
A chunk is a piece of a document, usually a few paragraphs.
An embedding is a list of numbers that represents the meaning of a piece of text.
An index (often a vector database) stores the embeddings so the closest ones can be found quickly.
The retriever is the part that takes a question and returns the best chunks.
The context is the set of retrieved chunks placed in the prompt.

Why it matters

Each later module is about one of these words. Knowing which word a problem belongs to is most of debugging: a wrong answer is a document problem (the fact was never there), a chunk problem (the fact was cut in half), a retriever problem (the right chunk was not found) or a generation problem (it was found and ignored).

In the real world: In a design review, an engineer says 'the retriever returned the right document but the wrong chunk', and everyone knows the fix lies in how the files were cut, not in the model.

Visual example

Vocabulary

Follow one help article from a file to an answer.

Six words, in the order data flows through them

word	plain meaning	in the Kirana Kart example
Document	One source file	The 'Refunds' help article
Chunk	A piece of a document	The paragraph on late orders
Embedding	Text turned into a list of numbers	A vector for that paragraph
Index	Where vectors are stored and searched	All help-centre vectors
Retriever	Finds the best chunks for a question	Returns the late-order paragraph
Context	Retrieved chunks placed in the prompt	What the model reads before answering

Using 'document' and 'chunk' as if they were the same. Search works on chunks, not documents. Saying 'we retrieve the top 5 documents' when you mean 5 chunks hides the most important design choice in the system, which is how the documents were cut.

Practise it: Design a Document Intelligence Platform, Design a Semantic Search System

RAG versus fine-tuning

What it is

Fine-tuning means training the model further on your own examples, which changes its weights.
RAG leaves the model alone and changes what it reads.

Fine-tuning teaches a model how to behave: a tone, a format, a specialised task. RAG gives a model what to know: facts that are private, large in number or often changing.

Why it matters

This is the most asked RAG interview question.

The short answer: use RAG for knowledge, fine-tuning for behaviour, and both when you need both.
RAG wins on facts because a document can be updated in minutes, a fact can be traced to its source, and access can be controlled per user.
A fine-tuned model has facts baked in: updating one means retraining, nothing can be cited, and every user gets the same knowledge.

In the real world: A legal-tech firm uses RAG to fetch the relevant clauses, and separately fine-tunes a small model to write in the firm's house style. Knowledge from retrieval, style from training.

Visual example

Comparison

They solve different problems. Knowledge goes to RAG, behaviour goes to fine-tuning.

RAG versus fine-tuning

question	RAG	fine-tuning
What changes?	The prompt	The model's weights
Best for	Facts that are private or change	Style, format, a narrow skill
Updating a fact	Edit the document, re-index	Retrain the model
Can cite sources?	Yes	No
Per-user access control?	Yes, filter what is retrieved	No, knowledge is baked in
Up-front cost	Build a search pipeline	Collect examples and train
Cost per question	Higher (longer prompts)	Lower (short prompts)

Fine-tuning a model on your documents and expecting it to recite them accurately. Models absorb style and patterns from fine-tuning far more reliably than specific facts, and they will still state wrong details with confidence.

RAG versus pasting everything into a long prompt

What it is

Modern models accept very long prompts, hundreds of pages of text. So why search at all? Why not paste the entire help centre into every prompt?

For a small, stable set of documents, that is a perfectly good design.
It stops working as the collection grows.

Why it matters

Four reasons, worth being able to list.

Size: a real company's documents are far bigger than any prompt.
Cost and speed: you pay for, and wait for, every word in the prompt on every question.
Accuracy: models find a fact less reliably when it is buried in a very long prompt.
Access control: with everything in the prompt, every user can be shown everything.

Retrieval sends only the few passages that matter, and only those the user may see.

In the real world: A startup with a 20-page handbook pastes the whole thing into the prompt and ships in a day. A hospital with two million pages of clinical guidance has no choice but to retrieve.

Declaring 'long context windows have killed RAG'. They raise the point at which you need retrieval; they do not remove it. The honest interview answer is a threshold: if everything fits comfortably and rarely changes, skip retrieval; otherwise retrieve.

When RAG is the wrong tool

What it is

RAG answers questions whose answer sits in a few passages of text. It is a poor fit when:

the answer needs every document ("how many complaints mentioned late delivery last month?");
it needs calculation over tables (use a database query);
the task is about style or skill rather than facts (use prompting or fine-tuning);
the model already knows the answer (general knowledge needs no retrieval).

Why it matters

Recognising the wrong fit is what separates a senior answer from a junior one. A retriever returns the top few chunks, so a question that needs all ten thousand tickets will be answered from five of them, confidently and wrongly.

In the real world: An operations manager asks the support assistant for last month's refund total; a well-designed system routes that to the orders database instead of searching help articles.

Answering counting and totalling questions with plain RAG. "How many" and "what is the average" are database questions. The model sees a handful of chunks and guesses a number.

A complete RAG system in thirty lines

What it is

Stripped of everything optional, RAG is four steps of code: embed the chunks, embed the question, find the closest chunks, and ask the model with those chunks in the prompt. The example uses the open-source sentence-transformers library for embeddings and a placeholder llm() function for whichever model you call.

Why it matters

Seeing the whole thing at once shows how little magic there is, and gives every later module a place to attach: chunking improves chunks, a vector database replaces the @ line, reranking reorders top, and prompt design improves prompt.

In the real world: A hackathon team gets a working document chatbot in an afternoon with code like this, then spends the next three months on the parts it leaves out: parsing, chunking, evaluation and access control.

import numpy as np
from sentence_transformers import SentenceTransformer

chunks = [
    "If your order arrives more than 30 minutes late, "
    "we refund the full delivery fee to your wallet.",
    "Report a damaged item within 48 hours with a photo "
    "and we refund the item price.",
    "The delivery fee is 25 rupees for orders under 199 rupees "
    "and free above that.",
]

# --- indexing (done once) ---
model = SentenceTransformer("all-MiniLM-L6-v2")
chunk_vecs = model.encode(chunks, normalize_embeddings=True)   # shape (3, 384)

# --- answering (done per question) ---
question = "My order was 40 minutes late. Do I get anything back?"
q_vec = model.encode([question], normalize_embeddings=True)[0]

scores = chunk_vecs @ q_vec               # similarity to every chunk
top = np.argsort(-scores)[:2]             # the 2 closest chunks
context = "\n\n".join(chunks[i] for i in top)

prompt = f"""Answer using only the context.
If the answer is not there, say you don't know.

Context:
{context}

Question: {question}"""

answer = llm(prompt)                      # any chat model

Stopping here. This works in a demo with three chunks and fails quietly with three hundred thousand. Every later module exists because one of these lines breaks at real scale or on real questions.