RAG evaluation

The bot is confidently wrong

A support assistant answers “How do I get a refund?” with a fluent paragraph — and quietly invents a “$10 credit” your docs never promised. In the interview, the question is not “is the answer good?” but: where is the bug — retrieval or generation?

Predict before reading: name the two distinct places this pipeline could have failed, and how you’d tell them apart.

Two systems in a trenchcoat

RAG is a pipeline: a retriever fetches candidate chunks, a generator writes an answer from them. Evaluate each stage on its own terms:

• Retrieval is a ranking problem (Chapter 2’s cousin): did the right chunks come back, and near the top? Graded with precision@k, recall@k, NDCG, MRR.
• Generation is a grounding problem: did the answer use the retrieved context faithfully, and actually address the question? Graded with faithfulness, answer relevance, and context precision — usually via an LLM judge (Chapter 7).

Decomposing this way is the whole skill: it turns “the bot is wrong” into “retrieval recall@5 is 0.6, so 40% of answerable questions never get the chunk” or “retrieval is fine but faithfulness is 0.7 — the generator is adding things.”

Grading retrieval

Retrieval returns a ranked list; you have a set of relevant chunk ids. The metrics from Chapter 2 carry over, now indexed by a cutoff k (how many chunks you feed the generator) — all in mini_eval:

from mini_eval import precision_at_k, recall_at_k, ndcg_at_k

precision_at_k(retrieved, relevant, k=5)  # of the top 5, how many are relevant
recall_at_k(retrieved, relevant, k=5)     # of all relevant, how many made the top 5
ndcg_at_k(retrieved, relevant, k=5)       # rewards relevant chunks ranked *higher*

The same precision/recall tension as Chapter 2 reappears over k: a bigger context window raises recall (more relevant chunks make the cut) but lowers precision (more distractors come with them) — and, as we’ll see, distractors are exactly what trigger the second kind of hallucination. NDCG adds what plain recall ignores — that a relevant chunk at rank 1 is worth more than the same chunk at rank 8:

Grading the answer

For the generated answer, the workhorse is faithfulness: break the answer into atomic claims and check each one is supported by the retrieved context. The score is the fraction grounded. Two siblings complete the picture (the core RAGAS metrics):

• Faithfulness — are the answer’s claims grounded in the context? (catches hallucination)
• Answer relevance — does the answer actually address the question? (catches evasive or off-topic answers)
• Context precision/recall — of the retrieved context, how much was needed, and was all the needed context retrieved? (connects back to the retriever)

Faithfulness is itself an LLM-as-judge task, so everything from Chapter 7 applies — biases, swap-and-aggregate, a human-anchored set.

See it

Two tabs: drag k to watch retrieval’s precision/recall/NDCG trade off, then switch to Faithfulness to see the answer’s claims checked against the context. Predict: on the retrieval tab, as you raise k from 1 to 10, what happens to precision versus recall?

Now explain the two screens together. Recall climbs with k (you eventually catch all the relevant chunks) while precision sags (distractors pile in); the faithfulness tab shows the generator inventing the “$10 credit” — a claim grounded in nothing retrieved.

Two hallucinations, opposite fixes

How this is graded

• Technical Correctness — you separate retrieval from generation and name the right metric for each (ranked-list vs grounding), not “we use RAGAS”.
• Trade-off Awareness — you reason about the precision/recall-over-k tension and how it couples to the two hallucination types.
• Evaluation Rigor — the weighted dimension: you’d build a small labeled set (relevant chunks per query + a faithfulness rubric), measure each stage, and localize failures rather than scoring end-to-end.
• Communication — “recall@5 is 0.6 so 40% of questions never get the chunk — that’s our ceiling, fix retrieval first” beats “the answers feel off”.

Industry variation

• Customer support / docs — faithfulness is paramount; a fluent unsupported answer is a trust and liability risk.
• Legal / medical — Type-II (right-sounding, wrong source) is the feared failure; provenance and citation-level grounding are required.
• Enterprise search — retrieval recall over a huge, permissioned corpus dominates; access-control leakage is its own eval.

Stretch: you don’t have relevance labels

Everything above assumed you knew which chunks were relevant. On a fresh corpus you don’t. How would you bootstrap a retrieval eval set without hand-labeling thousands of query–chunk pairs? (Hints: LLM-generated question–answer pairs from known chunks give you free relevance labels; context precision/recall can be judged by an LLM without a gold set; and production click/citation logs are weak labels. Each is reference-free — Chapter 6 — and each has its own biases to watch.)