Evaluating RAG: the builder’s loop

”B felt better”

Chapter 4 ended with two candidate fixes: pipeline A keeps the old tight budget and no floor; pipeline B adds a similarity floor and triples the budget. You read ten answers from each. B felt better. You wrote “B looks good” in the PR description.

Your reviewer asks three questions you can’t answer: Better on what? How often does each one answer when it shouldn’t? Which questions does B get wrong that A gets right? The honest reply to all three is “I don’t know” — you sampled ten happy-path questions and a feeling.

Predict: what’s the smallest artifact that would let you answer all three questions mechanically — and roughly how long would it take to build?

The golden set: your corpus, your exam

A golden set for RAG is a versioned file of rows like:

{"q": "Are gift cards refundable?",
 "relevant": [4],                  # chunk ids that answer it
 "fact": "non-refundable",         # substring the answer must carry
 "reachable": True}                # our retriever *should* find this

Three design rules separate a useful set from a vanity one:

1. Cover the edge classes, not just the FAQ. For every handful of plainly answerable questions, include: a paraphrase row (answerable by the corpus, phrased like a user — “Can I get my money back?”), a trap row (morphology, near-miss vocabulary — “Can I exchange a clearance item?”), and abstain rows — out-of-scope questions whose only correct behavior is “I don’t know.” Abstain rows are the most commonly omitted and the cheapest to write, and without them you cannot measure the difference between a system that knows its limits and one that answers everything.
2. Label behavior, not prose. A relevant-chunk id set plus a must-contain fact is enough to score mechanically; a hand-written “ideal answer” invites string-matching theater. (Bootstraps when labels are expensive: generate questions from known chunks — free relevance labels — and mine real user queries from logs; the Evaluation guide’s Chapter 9 stretch covers the biases each shortcut buys.)
3. Small, versioned, and in CI. Thirty to a hundred rows, next to the code, re-run on every retrieval-affecting change — chunking config, embedding model, floor, budget. The set is to your pipeline what the test suite is to your code; “we’ll label a thousand examples next quarter” is how teams end up with zero.

Two halves, one dashboard

Chapter 4’s trace makes the split mechanical. The retrieval half is a ranking problem, and mini_eval (the Evaluation guide’s companion) already speaks ranked lists — this is where the two guides shake hands:

from mini_eval import recall_at_k, precision_at_k     # measurement (guide 1)
from mini_rag import RagPipeline                       # mechanism (this guide)

trace = pipe.run(row["q"])
included = [h.index for h in trace.included]           # what the model SAW
recall_at_k(included, set(row["relevant"]), k=len(included))   # context recall

Score the included context, not just the raw hits — Chapter 4 taught why: a chunk retrieved at rank #3 and cut by the budget never happened, as far as the generator is concerned.

The generation half is a grounding problem: faithfulness (are the answer’s claims supported by the context?), answer relevance, and the Type-I/Type-II hallucination split — all built and graded in the Evaluation guide’s Chapter 9; this chapter doesn’t repeat it. One builder’s note belongs here: our companion generator is faithful by construction (it can only quote), so its faithfulness column is trivially green. The moment you swap in a real LLM, faithfulness becomes a measured number with an LLM-as-judge behind it (RAGAS is the production bridge) — budget for it the day you swap, not after the first fabricated refund policy ships.

Read the halves together: high faithfulness + low context recall means the generator is honestly summarizing the wrong inputs; the fix is upstream, and no prompt work will move it.

From scores to a decision

Aggregate scores hide the texture a decision needs. Categorize each golden-set row’s outcome instead — these map one-to-one onto Chapter 4’s failure points:

| Outcome | Meaning | Failure point | |---|---|---| | correct | answered, fact present | — | | junk | answered when it should have abstained | FP1 + missing floor | | retrieval-miss | needed chunk never ranked | FP1 | | lost-in-assembly | retrieved, then cut by budget or floor | FP2 | | answer-extraction miss | fact in context, answer missed it | FP3 / FP4 | | abstain (safe) | declined a question the corpus can’t answer | working as designed | | abstain (unreachable) | declined one the corpus answers but this retriever can’t reach | FP1 — fix retrieval, not the floor | | abstain (missed) | declined a reachable, answerable one | floor/threshold too high |

Then the decision stops being aesthetic. Coverage (how often it answers) and harm (how often it answers wrongly) move in opposite directions as you raise floors and thresholds — choosing the operating point is the same threshold-on-a-curve judgment as the Evaluation guide’s Chapter 2, applied to a pipeline. A support product usually pays more for one confident wrong answer than for three honest abstentions; an internal research tool may price it the other way. The golden set doesn’t make that call for you — it makes the call visible.

See it: the table that settles the argument

Config A (“ship-it”) versus config B (“hardened”), six golden rows, every cell a real pipeline run. Predict before revealing: which config wins, and is there a question where both lose? Expand the rows — especially the money-timing question, where the same failure shows up as a different category in each config.

How this is graded

• Technical Correctness — you score the included context (what the model saw), not raw retrieval, and you know why the toy’s faithfulness is trivially 1.0 while a real generator’s isn’t.
• Trade-off Awareness — coverage vs harm as an explicit operating point; the floor as a curve you choose a point on, priced per product.
• Evaluation Rigor — this chapter’s weighted dimension: a versioned golden set with abstain rows, two-half measurement, outcome categories, and re-runs on every config change. “I read ten answers” never appears.
• Communication — “B converts one junk answer into an honest abstain and loses nothing; both configs still fail the morphology trap — that’s a ranking problem, here’s the table” beats “B felt better.”

Industry variation

• Customer support / consumer — junk is the expensive cell: one invented policy answer costs more trust than ten abstains; teams bias the operating point hard toward safety and instrument the abstain UX.
• Legal / medical / compliance — the golden set is itself an audited artifact: who labeled it, when, against which corpus version; provenance of the labels matters as much as the scores.
• Startups — the six-row table in this chapter is an afternoon, runs in CI in a minute, and already beats most teams’ RAG QA. Build it before the demo, not after the incident.

Stretch: every failure points the same direction

Look back at the table: the morphology trap defeated both configs, the paraphrase forced a choice between junk and abstaining, and the money-timing question failed because a chattier chunk outranked the right one. Three different categories — and one common cause: the ranking your pipeline feeds everything downstream is lexically brittle. No budget, floor, or prompt fixes ranking. What stage would you add or upgrade so that “money back” finds the refund policy, “clearance item” beats the chunk that merely says “item” twice, and the right chunk reaches rank #1 more often? That’s Chapter 6 — hybrid retrieval, rerankers, and query rewriting: the upgrades that attack ranking quality itself.