RAG · Module 2 of 14 · 10 concepts · about 15 min

Embeddings for Retrieval

How text becomes a list of numbers, how closeness between two lists is measured, and how to choose the model that produces them. Everything in retrieval rests on this.

At work: The engine under every semantic search: matching a customer's wording to a help article, a job description to a CV, a bug report to a similar past ticket.

What an embedding is: meaning as a list of numbers

What it is

An embedding model reads a piece of text and outputs a fixed-length list of numbers, called a vector or an embedding.

A typical one has 384, 768 or 1,536 numbers.
The model is trained so that texts with similar meaning get vectors that are close together, and unrelated texts get vectors that are far apart.
"I want my money back" and "how do I get a refund" share almost no words, yet their vectors land close to each other.

Why it matters

Computers cannot compare meanings, but they can compare numbers very quickly. Embeddings turn 'find the passage that means the same as this question' into 'find the nearest list of numbers', which is a problem with fast, well-understood solutions.

In the real world: A travel site lets customers search 'somewhere warm in December with kids' and matches it to hotel descriptions that never use those words.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

vecs = model.encode([
    "I want my money back",
    "How do I get a refund?",
    "What time do you open?",
])
vecs.shape        # (3, 384): three texts, 384 numbers each

sims = model.similarity(vecs, vecs)
# the first two texts score high with each other,
# the third scores low against both

Trying to read meaning into individual numbers. Position 17 is not 'refund-ness'. The meaning lives in the whole vector and in its direction relative to other vectors, never in a single entry. (The diagrams in this module use made-up three-number vectors with readable meanings only so the arithmetic can be followed.)

Practise it: Design a Semantic Search System

Cosine similarity, dot product and Euclidean distance

What it is

Three ways to measure how close two vectors are.

Cosine similarity measures the angle between them: 1 means same direction, 0 means unrelated, and it ignores length.
Dot product multiplies matching entries and adds them up: it rewards both a similar direction and a long vector.
Euclidean distance is the straight-line distance between the two points: smaller means closer.

Why it matters

Cosine similarity is the default in RAG, because the direction of an embedding carries its meaning while its length mostly does not.

The formula is cos(a, b) = (a · b) / (|a| × |b|): the dot product divided by the two lengths.
An interviewer will often ask you to compute one by hand, so work through a row of the table below.

In the real world: A recruitment platform ranks CVs against a job post by cosine similarity, so a long CV does not outrank a short, well-matched one just for being long.

Visual example

Computed

One query scored against all six chunks. Late order refund wins on every measure, though the question shares no keywords with it.

q = [0.8, 0.7, 0] (toy vectors: refunds, delivery, membership)

chunk	vector	cosine ↑	dot ↑	distance ↓
c1 Late order refund	[0.9, 0.6, 0]	0.99	1.14	0.14
c2 Damaged item refund	[0.9, 0.1, 0]	0.82	0.79	0.61
c3 Delivery fee	[0.1, 0.9, 0.2]	0.72	0.71	0.75
c4 Kart Plus membership	[0, 0.5, 0.9]	0.32	0.35	1.22
c5 Change address	[0, 0.8, 0]	0.66	0.56	0.81
c6 Error KK-4471	[0, 0.1, 0.3]	0.21	0.07	1.04

↑ higher is closer, ↓ lower is closer. Real vectors have hundreds of numbers; the arithmetic is identical.

Query “Can I get my money back if the rider is delayed?” as a toy vector q = [0.8, 0.7, 0], against the Late order refund chunk c1 = [0.9, 0.6, 0].

dot product: 0.8×0.9 + 0.7×0.6 + 0×0 = 1.14
length of q: √(0.8² + 0.7² + 0²) = 1.063
length of c1: √(0.9² + 0.6² + 0²) = 1.082
cosine: 1.14 / (1.063 × 1.082) = 0.99

Using a different measure at search time from the one the embedding model was trained with. Each model's documentation names its measure. Mixing them does not raise an error; it just quietly returns worse results.

Normalised vectors: why the three measures agree

What it is

To normalise a vector is to divide it by its own length, so its length becomes exactly 1. Once every vector has length 1, cosine similarity and dot product give the same number, and ranking by Euclidean distance gives the same order as ranking by cosine.

Why it matters

This is why the choice of measure matters less than it first seems.

Most embedding models output normalised vectors, or you normalise them yourself, and then you can use the dot product, which is the cheapest to compute.
Vector databases are fastest with dot product for exactly this reason.

In the real world: A team switches its index from cosine to dot product to cut search time, and the results are unchanged because the vectors were normalised at indexing time.

import numpy as np

v = np.array([0.9, 0.6, 0.0])
unit = v / np.linalg.norm(v)
np.linalg.norm(unit)                 # 1.0

# with unit vectors: cosine == dot product
a, b = unit, np.array([0.8, 0.7, 0.0]) / np.linalg.norm([0.8, 0.7, 0.0])
a @ b                                # 0.99, the cosine from the table

Using dot product on vectors that are not normalised. Longer vectors then win regardless of meaning, and long chunks crowd out short, more relevant ones.

Bi-encoders: why queries and chunks are embedded separately

What it is

The embedding model used for retrieval is called a bi-encoder: it embeds the query on its own and each chunk on its own, and only afterwards compares the two vectors. The query and the chunk never see each other inside the model.

Why it matters

This is what makes search fast.

Because chunks are embedded without knowing the question, all of them can be embedded ahead of time and stored.
At question time only one new embedding is needed, for the query, followed by a nearest-vector search.
The cost is some accuracy: a model that reads the query and chunk together (a cross-encoder, in the Reranking module) judges relevance better but cannot be precomputed.

In the real world: A news archive embeds forty million articles once, over a weekend, and then answers each reader's search by embedding only the eight words they typed.

Expecting a bi-encoder to catch fine distinctions. Squeezing a whole chunk into one vector loses detail, so "refund for late orders" and "no refund for late orders" end up close together. Fast and roughly right is the bi-encoder's job; precision comes from reranking.

Query and document prefixes (asymmetric search)

What it is

A question is short ("late order refund?"); the passage that answers it is long and worded differently.

This is asymmetric search.
Many embedding models are trained for it and expect you to tell them which side they are embedding: by adding a prefix such as query: or passage: to the text, or by passing a setting such as an input type of 'query' or 'document'.

Why it matters

The model learned to place a question near its answer, not near other questions, and it uses the prefix to know which role the text plays. Leave it out and retrieval quality drops by a noticeable margin, with no error to tell you.

In the real world: A support team sees search quality jump after a one-line fix: the indexing job had been embedding articles with the 'query' setting instead of the 'document' one.

# The exact prefix or setting is model-specific: read the model card.
# For the E5 family it is a literal text prefix:

chunk_vecs = model.encode(["passage: " + c for c in chunks])
q_vec = model.encode("query: " + question)

Copying embedding code from a tutorial written for a different model. Prefix conventions differ between model families, and some need none. Always check the model's own documentation.

Practise it: Design a Semantic Search System, Design a Production RAG System, Design an Enterprise Knowledge Assistant

Choosing an embedding model

What it is

Five things decide the choice.

Quality on your kind of text: public leaderboards such as MTEB rank models on retrieval tasks, which is a starting point, not a verdict.
Maximum input length: text beyond the limit (often 512 tokens, sometimes 8,000 or more) is silently cut off.
Languages: an English-only model fails on Hindi.
Dimensions: more numbers per vector means more storage and slower search.
Hosted or self-hosted: an API is simple to start with; your own model keeps data in-house and has no per-call fee.

Why it matters

The embedding model fixes the ceiling on retrieval quality, and it is expensive to change later, because changing it means re-embedding every chunk. So it deserves a real test: take fifty real questions with known answers, run two or three candidate models, and compare how often the right chunk comes back.

In the real world: A marketplace serving both Hindi and English buyers tests three multilingual models on real search logs and picks the second-ranked one on the leaderboard, because it was best on mixed-language queries.

Visual example

Checklist

The leaderboard score is one row out of six.

What to check before choosing an embedding model

question	why it matters
How does it do on my own test questions?	The only score that predicts your results
What is the maximum input length?	Longer chunks are cut off without warning
Does it cover my languages?	English-only models fail on everything else
How many dimensions?	Drives storage cost and search speed
Hosted API or self-hosted?	Cost per call versus running your own servers; data privacy
Does it need query/document prefixes?	Getting this wrong quietly lowers quality

Picking the top model on a leaderboard without testing it on your own data. Leaderboards measure general text; a model that is best on Wikipedia can be mediocre on medical notes, legal clauses or code.

Embedding dimensions and storage cost

What it is

Each number in a vector is normally stored as a 4-byte decimal.

So one vector takes dimensions × 4 bytes, and an index takes chunks × dimensions × 4 bytes before any overhead.
Some newer models are trained so that their vectors can be shortened (keeping only the first 256 or 512 numbers) with a small loss in quality; this is called Matryoshka embedding, after the nesting dolls.

Why it matters

Dimension is a cost you can calculate before building anything, and interviewers like to see you do it.

Fast vector search usually keeps vectors in memory, so the table below is roughly the RAM bill.
Doubling the dimensions doubles it, and rarely doubles the quality.

In the real world: A company with 100 million chunks cuts its vectors from 1,536 to 512 numbers, loses about a point of retrieval quality, and removes two-thirds of its vector memory bill.

Visual example

Computed

Raw vector storage: chunks × dimensions × 4 bytes. Same data, very different bills.

Storage in GB for float32 vectors, before index overhead

dims	bytes each	typical of	1M chunks	100M chunks
384	1536	small open model	1.5	153.6
768	3072	mid-size model	3.1	307.2
1536	6144	large hosted	6.1	614.4
3072	12288	largest hosted	12.3	1228.8

One row worked out

per vector: 1,536 × 4 bytes = 6,144 bytes

100M chunks: 100,000,000 × 6,144 bytes = 614.4 GB

Assuming more dimensions always means better search. Past a few hundred, gains are small and task-dependent, while cost grows in a straight line. Measure the quality difference before paying for it.

One model for indexing and querying

What it is

The query must be embedded with exactly the same model, version and settings that embedded the chunks. Vectors from two different models live in two unrelated number spaces; comparing them is like measuring one distance in kilometres and another in degrees Celsius.

Why it matters

It makes the embedding model a long-term commitment.

Switching models, or even upgrading a version, means re-embedding the whole collection and rebuilding the index.
Good systems record the model name and version beside every vector, and build the new index next to the old one before switching over.

In the real world: A hosted embedding API retires an old model; the team that stored the model version with each vector re-indexes in a planned weekend, and the team that did not finds out when search breaks.

Upgrading the embedding model in the query service only. Nothing crashes. Every search simply returns near-random chunks, and the first sign is users complaining the assistant has 'got worse'.

Fine-tuning an embedding model

What it is

A general embedding model can be trained further on your own data.

The training examples are pairs: a real question and the chunk that answers it (a positive), plus chunks that look related but do not answer it (hard negatives).
Training pulls the question's vector towards the positive and pushes it away from the negatives. This is called contrastive learning.

Why it matters

It helps most when your vocabulary is specialised: part numbers, drug names, internal jargon that a general model has never seen used.

A few thousand good pairs can lift retrieval noticeably.
It is also a late step: try better chunking, hybrid search and reranking first, because they are cheaper and need no training data.

In the real world: An industrial supplier fine-tunes on past search logs so that 'M8 hex 40mm' lands on the right bolt, something a general-purpose model treated as noise.

Training with easy negatives only (random chunks). The model already tells a refund article from a recipe. It learns nothing unless the negatives are the confusing ones: the damaged item refund chunk when the question was about late order refunds.

What embeddings are bad at

What it is

Embeddings capture general meaning and blur specifics. They are weak on:

exact identifiers (order IDs, error codes like KK-4471, product codes);
numbers and dates ("under ₹199" and "under ₹999" look alike);
negation ("refundable" versus "not refundable");
rare words the model seldom saw in training.

Why it matters

Knowing the weak spots tells you what to add:

Exact identifiers are what keyword search is good at, which is why production systems combine the two (hybrid retrieval).
Numbers and dates belong in metadata filters.
Fine distinctions such as negation are what a reranker is for.

In the real world: An electronics retailer finds that searches for an exact model number return similar-sounding products; adding keyword search alongside embeddings fixes it within a day.

Blaming the language model for a wrong answer when the retriever fetched a near-miss. A chunk about ₹999 orders retrieved for a ₹199 question produces a fluent, wrong answer, and the fault lies in the search step.

Practise it: Design a Semantic Search System, Design a Production RAG System