Retrieval 101 — embeddings and vector search

The spec that couldn’t save you

Chapter 1 left you with a hardened support assistant: a pinned spec, a validated output contract, a held-out test set. A customer asks “Can I get my money back?” and the assistant replies — exactly as designed — “I don’t know.”

The refund policy is sitting right there in policies/refunds.md. You read the logs: the context the assistant received contained the shipping FAQ and two articles about gift cards. The model never saw the refund policy, so the missing-context rule fired. The prompt did its job. The layer that chose the context — keyword search over your help center — found nothing, because the policy says “refund” and the customer said “money back.”

Predict: what, specifically, would have to change so this query finds that document? Better prompt? Bigger model? Something else entirely?

Search is “vectorize, then measure angles”

Every vector search system — from the toy you’re about to build to a billion-vector production cluster — is the same three steps:

1. Vectorize every document once, and the query at search time.
2. Score the query vector against each document vector — usually cosine similarity.
3. Top-k: return the k best scores.

The honest starting vectorizer is TF-IDF, because you can read every line of it. A term that appears often in this document (TF) and rarely across documents (IDF) gets a big weight:

where $N$ is the corpus size and $\mathrm{df}_t$ counts documents containing term $t$. Cosine measures the angle between the sparse vectors, so a long document can’t win just by being long. In mini_rag (the companion you’ll grow through this guide), the whole index is:

class TfidfIndex:
    """A tiny in-memory TF-IDF index — the "vector store" of a RAG system,
    minus the scale."""

    def __init__(self, corpus: list[str]) -> None:
        self.docs = list(corpus)
        corpus_tokens = [tokenize(d) for d in self.docs]
        self.idf = idf(corpus_tokens)
        self._vectors = [tfidf_vector(t, self.idf) for t in corpus_tokens]

    def search(self, query: str, k: int = 5) -> list[Hit]:
        qvec = tfidf_vector(tokenize(query), self.idf)
        scored = [Hit(i, cosine_similarity(qvec, dvec), self.docs[i])
                  for i, dvec in enumerate(self._vectors)]
        scored = [h for h in scored if h.score > 0.0]
        scored.sort(key=lambda h: (-h.score, h.index))
        return scored[:k]

Worked example, small enough to check by hand. Three documents:

• A — “You may request a refund within 30 days.”
• B — “Our support hours are 9am to 5pm.”
• C — “Refund requests need the order number.”

Query: “refund order”. With $N=3$: “refund” appears in two documents, so $\mathrm{idf} = \ln(4/3)+1 \approx 1.29$; “order” appears in one, so $\ln(4/2)+1 \approx 1.69$ — the rarer term carries more weight. Score each document: C shares both terms and wins with $\cos \approx 0.53$; A shares only “refund” and scores $\approx 0.17$; B shares nothing and scores exactly 0. Run TfidfIndex(corpus).search("refund order") in the companion and you’ll get these numbers.

Now trace one yourself before moving on: same corpus, query “support hours”. Which document wins, which two scores are exactly zero, and why do “support” and “hours” carry equal weight here? (You have everything you need: count document frequencies, then think about angles.)

When words don’t match meaning

Run the opener’s query against a real support corpus and the failure is exact, not vague. “Can I get my money back?” shares zero tokens with “You may request a refund within 30 days of purchase.” — look at tfidf_vector: a query term that isn’t in the corpus vocabulary contributes nothing, so the score is literally 0.0, and search returns an empty list. The relevant document wasn’t outranked; it was invisible.

Lexical retrieval misses in two characteristic ways:

• Paraphrase — “money back” vs “refund”: same meaning, disjoint spelling.
• Morphology — “ship” vs “shipping”, “refund” vs “refundable”: our tokenizer does no stemming, so near-identical words are different terms. (Stemming patches the easy cases and does nothing for paraphrase.)

Dense embeddings fix this by replacing the vectorizer — and only the vectorizer. An embedding model is a neural encoder trained so that texts that mean similar things land near each other in a ~1,000-dimensional space: “money back” and “refund policy” end up close because the model learned from how language is used, not how it’s spelled. The search half is unchanged — vectorize, cosine, top-k. That’s why you built the TF-IDF version: it’s the same skeleton with a swappable first step.

The trade runs both directions, though:

| | Sparse (TF-IDF / BM25) | Dense (embeddings) | |---|---|---| | Paraphrase, concepts | misses | catches | | Exact rare tokens — error codes, SKUs, names | exact match | can blur | | Cost to build | trivial, no model | embedding model per doc + query | | Explainability | inspect shared terms | opaque similarity |

Production systems usually run both and merge the rankings — hybrid search. That, plus reranking the candidates with a stronger model, is Chapter 6; here it’s enough to know neither side dominates. When you pick an embedding model, frame it by constraints — dimensionality drives storage, per-query latency, and cost; a leaderboard (MTEB) tells you about benchmark text, not your domain’s acronyms — “I’d benchmark a 1,024-dim open model against the API default on our queries” beats naming this month’s winner.

From a list to a hundred million vectors

mini_rag scores the query against every document — exact search, $O(n \cdot d)$ per query. Honest advice: this is fine for longer than you’d guess. At 100K vectors it’s milliseconds; our toy’s real sin at scale is rebuilding the index per call, not the linear scan.

When latency does break, approximate nearest neighbor (ANN) indexes trade a little correctness for a lot of speed. The workhorse is HNSW: a layered graph where sparse upper layers make long hops and dense lower layers refine locally; search greedily descends in roughly $O(\log n)$. The price is in the name — approximate: the true best match is sometimes not returned. That price has a metric, recall@k — the fraction of the true top-k your index actually returns — and you measure it against a golden set, then tune (HNSW’s ef_search buys recall with latency). Accepting an ANN index without measuring recall@k is shipping a known unknown.

Storage is the other budget, and it’s arithmetic, not folklore. Half a million 10-page documents chunk to roughly 4.5M chunks; at 1,024 dimensions of float32:

— fits in RAM on one node, and that calculation (not the leaderboard) is what should pick your embedding dimension.

Where the vectors live is a constraints question:

• Already run Postgres? pgvector adds a vector column + HNSW index to the database you operate today — no new system. The right default below ~5–10M vectors.
• Dedicated vector DB (Qdrant, Weaviate, Pinecone, …) when you need what Postgres can’t give: managed scale, built-in hybrid search, sub-10ms ANN at high QPS.
• Access control lives in the store, not the app: filter by tenant_id / project_id inside the search query. Filtering after retrieval means the wrong tenant’s chunks existed in a candidate list — a security incident, not a relevance bug.

See it: four queries, one index

The explorer below runs the corpus through mini_rag exactly as built above. For each query, predict before revealing: which document ranks #1, and how many match at all? Then slide k and watch precision@k and recall@k fight. The paraphrase and morphology tabs are the lexical misses from this chapter — explain each ranking using only “shared tokens × rarity” and you’ve got the mechanism.

How this is graded

• Technical Correctness — you can trace TF-IDF and cosine by hand, and you say precisely what changes when moving to embeddings: the vectorizer, not the search.
• Trade-off Awareness — this chapter’s weighted dimension: sparse vs dense by content type, exact vs ANN by scale, pgvector vs dedicated store by team constraints — each named with its cost.
• Evaluation Rigor — you refuse “the results look right”: a golden set, recall@k against it, and re-measurement when the index or model changes.
• Communication — “30K vectors — exact search in pgvector, tenant filter in the query, recall@5 ≥ 0.9 on a golden set” beats a tour of vector databases.

Industry variation

• Startups — the whole layer is pgvector on the Postgres you already run; adding a dedicated vector DB before ~5M vectors is résumé-driven infrastructure.
• Enterprise — compartmented data makes store-level filtering non-negotiable (legal, healthcare, defense); “filter in the app” fails the security review.
• Latency-SLA shops (marketplaces, search products) — ANN parameters are tuned against p95 budgets, and recall@k is tracked per release like a test suite. Relatedly: “build a tiny vector store” is now a live-coding interview classic — normalize at index time, score in one pass, justify your top-k.

Stretch: the right document, the wrong slice

You swap in dense embeddings and the paraphrase miss disappears — recall on the golden set jumps. Yet refund answers still come back wrong: the chunk that matches “money back” ends with “…you may request a refund” and the 30-day limit sits in the next chunk, which didn’t make the top k. Nothing is wrong with the vectors. What single upstream decision — made before any embedding was computed — caused this, and what would you change about it? That decision is chunking, and it’s Chapter 3: how you cut documents determines what a vector can even represent.