Bezen

RAG · Module 3 of 14 · 6 concepts · about 10 min

Loading and Parsing Documents

Turning real files (PDFs, web pages, slides, scans) into clean text with useful labels attached. If this step produces rubbish, no later step can repair it.

At work: The data-engineering half of every RAG project: getting contracts, manuals, wikis and scanned forms out of their file formats and into text a search system can use.

The ingestion pipeline

What it is

Ingestion is everything that happens to a document before it can be searched: load it from where it lives (a shared drive, a wiki, a ticketing system), parse it into plain text, clean that text, attach metadata (labels such as title, date and owner), cut it into chunks, embed each chunk and store the result.

Why it matters

The model can only answer from what ingestion produced.

  • A table that was flattened into a jumble of numbers, or a page whose two columns were read straight across, becomes a chunk that no search will ever match and no model can read.
  • Teams routinely spend more time here than on anything involving the model, and it is usually where the largest quality gains are.

In the real world: An insurer's assistant gives wrong coverage answers until someone opens the parsed text and finds every policy table had been turned into a single unreadable line.

Visual example

Ingestion

Each stage can only be as good as the one before it.

From a file to a searchable chunk

  1. LoadFile, permissions, last-modified time
  2. ParseText, headings, tables
  3. CleanMenus, footers, duplicates out
  4. Add metadataTitle, section, date, who may see it
  5. ChunkCut into passages
  6. Embed and storeVector + text + metadata

Read the parsed text of ten documents with your own eyes before building anything else.

Treating ingestion as a one-off script. Documents are added, edited and deleted every day, so ingestion is a running service with a queue, retries and monitoring, not a notebook someone ran once.

Parsing PDFs, web pages and office files

What it is

Each format hides text differently.

  • A PDF stores characters at positions on a page, with no idea of paragraphs, columns or reading order, so a parser has to guess.
  • A web page wraps a little content in a lot of menus, cookie banners and footers.
  • Word and PowerPoint files have real structure (headings, slides, speaker notes) that is easy to read if the parser keeps it.

Why it matters

Structure is worth keeping.

  • A parser that outputs headings as headings (for example as Markdown: ## Refunds) lets the chunker cut at section boundaries and lets each chunk carry its section title.
  • A parser that outputs one undivided wall of text throws that away.

In the real world: A law firm switches to a layout-aware parser after finding that two-column court filings had been indexed with their columns interleaved line by line.

# Two very different results from the same PDF page.

# A plain text extractor: order and structure lost
#   'Refunds Late orders Damaged items If your order Report a damaged
#    arrives more than item within 48 hours 30 minutes late ...'

# A layout-aware parser that outputs Markdown: structure kept
#   '## Refunds
#    ### Late orders
#    If your order arrives more than 30 minutes late ...
#    ### Damaged items
#    Report a damaged item within 48 hours ...'

Choosing a parser by how easy it is to install. The first result above came from a two-column page read straight across. It contains every word and none of the meaning, and nothing downstream will flag it.

Tables, images and scanned pages

What it is

Three kinds of content defeat plain text extraction.

  • Scanned pages are pictures of text and need OCR (optical character recognition) to become text at all.
  • Tables lose their meaning when flattened, because a number is only meaningful beside its row and column labels.
  • Charts and diagrams contain information that is not text in the first place.

Why it matters

Much of a company's most valuable knowledge sits in exactly these: price tables, specification sheets, scanned contracts. The usual approaches are: OCR for scans; converting tables to Markdown or to one sentence per row so the labels stay attached; and asking a model that can read images to write a text description of each chart, which is then indexed like any other text.

In the real world: A hospital's assistant cannot answer dosage questions until the scanned drug charts are re-processed with table-aware OCR and spot-checked by a pharmacist.

Flattened table (useless):
  Order value Delivery fee Under 199 25 199 and above Free

Kept as Markdown (readable by search and by the model):
  | Order value   | Delivery fee |
  |---------------|--------------|
  | Under ₹199    | ₹25          |
  | ₹199 and above| Free         |

Or one sentence per row (best for search):
  For orders under ₹199 the delivery fee is ₹25.
  For orders of ₹199 and above delivery is free.

Trusting OCR output without checking it. OCR confuses 0 with O and 1 with l, and drops decimal points. In a price list or a dosage table, that is a wrong answer delivered with a citation.

Cleaning: boilerplate and duplicates

What it is

  • Boilerplate is text that repeats on every page and says nothing: navigation menus, 'Was this article helpful?', legal footers, page numbers, headers.
  • Duplicates are the same content stored more than once: a policy that exists as a PDF, a wiki page and three emailed copies, or last year's version beside this year's.

Why it matters

Both poison retrieval.

  • Boilerplate makes unrelated chunks look alike, since they share the same footer.
  • Duplicates are worse: if the top five results are five copies of one paragraph, the model sees one piece of evidence and the genuinely useful second and third passages are pushed out.
  • Old versions beside new ones produce confident answers from the outdated one.

In the real world: A company wiki assistant keeps quoting a retired leave policy because the old page was never deleted, only unlinked, and the crawler indexed both.

import hashlib

def fingerprint(text: str) -> str:
    """Same text (ignoring case and spacing) gives the same fingerprint."""
    normalised = " ".join(text.lower().split())
    return hashlib.sha256(normalised.encode()).hexdigest()

seen, unique = set(), []
for chunk in chunks:
    fp = fingerprint(chunk)
    if fp not in seen:
        seen.add(fp)
        unique.append(chunk)

Removing only exact duplicates. Two copies that differ by a date in the footer have different fingerprints. Strip the boilerplate first, then look for duplicates, and treat near-identical chunks (very high similarity) as duplicates too.

Metadata: the facts stored beside each chunk

What it is

Metadata is structured information kept alongside a chunk's text and vector: which document it came from, its title and section heading, the page number, the date it was last updated, its language, its type (policy, FAQ, contract), and who is allowed to read it.

Why it matters

Metadata does three jobs that embeddings cannot.

  • Filtering: 'only documents from 2025', 'only what this user may see'.
  • Citations: the source link and page number shown with the answer.
  • Context for the model: knowing a chunk is from 'Refund policy, updated March 2025' helps it weigh the passage.

Deciding what to capture has to happen at ingestion; it is very hard to add afterwards.

In the real world: A consultancy's assistant must never show one client's documents to another client's team; the client name stored on every chunk is what makes that filter possible.

Visual example

Metadata

Every field earns its place by enabling a filter, a citation or a fix.

What each metadata field is for
fieldused for
doc_id, chunk positionUpdating or deleting a document's chunks; fetching neighbours
title, sectionCitations; giving the model context
url, pageA link the user can click to check the answer
updated_atPreferring fresh content; filtering out old versions
language, doc typeFiltering the search to what is relevant
allowed_groupsAccess control: who may be shown this chunk
chunk = {
    "id": "refunds-2025#late-orders#0",
    "text": "If your order arrives more than 30 minutes late, ...",
    "vector": [...],
    "metadata": {
        "doc_id": "refunds-2025",
        "title": "Refund policy",
        "section": "Late orders",
        "url": "https://help.kiranakart.example/refunds#late",
        "updated_at": "2025-03-14",
        "language": "en",
        "allowed_groups": ["public"],
    },
}

Storing only text and vectors, then discovering in month three that you need to filter by date or restrict by department. That means re-ingesting everything. Capture the source, section, date and permissions from day one, even if nothing uses them yet.

Practise it: Design a Semantic Search System

Document identity and versions

What it is

Every document needs a stable ID that stays the same when its content changes, and every chunk needs an ID derived from it.

  • The pipeline also keeps a content hash: a short fingerprint of the text.
  • When a document is seen again, the hash says whether it has actually changed.

Why it matters

Without stable IDs, an edited document cannot replace its old chunks, so both versions sit in the index and the old one keeps being retrieved. With IDs and hashes, re-ingestion becomes cheap and safe: unchanged documents are skipped (no embedding cost), changed ones have their old chunks deleted and new ones written, and deleted documents have their chunks removed.

In the real world: A nightly sync job over 400,000 wiki pages finishes in minutes and costs almost nothing, because the hash check finds only the few hundred pages that changed.

Using the file name or a random ID as the identity. Rename the file and it becomes a 'new' document while the old chunks live on. Use the source system's own permanent ID (the wiki page ID, the database key).