TF-IDF From Scratch

Dense embeddings did not make lexical retrieval obsolete: every serious RAG system still runs BM25 or TF-IDF alongside them, because exact tokens like error codes and product SKUs are where embeddings quietly fail.

Implement tfidf(documents) taking a list of strings and returning (matrix, vocabulary):

• vocabulary is the sorted list of unique terms across all documents
• matrix has shape (n_documents, n_terms) with cell [i, j] = tf × idf

Definitions to use exactly:

• Tokenise by lowercasing and splitting on non-alphanumeric characters, discarding empties
• tf(t, d) = count of t in d divided by the total token count of d
• idf(t) = log(N / df(t)) where N is the document count and df(t) the number of documents containing t

A term appearing in every document gets idf = log(1) = 0 and contributes nothing. That is the property that makes the whole thing work.