Reference-based vs reference-free evaluation

Two answers, both correct, zero overlap

Your model answers “What’s the refund window?” with “You have thirty days to ask for your money back.” Your golden reference says “Refunds may be requested within 30 days of purchase.” Same meaning; almost no words in common. Your exact-match metric scores it 0. In the interview: was the model wrong, or was the metric?

Predict: name a task where comparing to a golden answer is the right call — and one where it actively misleads.

The spectrum

Grading methods sort by two questions: do you have a reference? and is the answer space closed (one right answer) or open (many)?

| Regime | Right tool | Example | | --- | --- | --- | | Reference, closed answer | exact match / verifier / unit tests | math answer, code that must pass tests, classification | | Reference, open answer | embedding similarity, multiple references, judge with reference | summarization, translation, long-form QA | | No reference | reference-free: LLM judge on a rubric, faithfulness, self-consistency | production logs, open chat, RAG answers |

The spectrum runs from cheap-and-exact to flexible-but-biased. Exact match and verifiers are free, deterministic, and bias-free — but only exist when correctness is checkable. LLM judges (Chapter 7) handle anything but bring cost and bias. Slide as far toward the cheap, exact end as the task allows, and no further.

Where each one breaks

• Exact match / BLEU / ROUGE — brittle on open-ended output: they reward lexical overlap, not meaning, so a correct paraphrase scores low and a fluent wrong answer that echoes the reference’s words scores high. Fine for closed answers; dangerous for open ones.
• Embedding similarity — forgives paraphrase, but can call a confidently wrong answer “similar” because it’s on-topic; it measures relatedness, not correctness.
• Reference-free LLM judge — handles anything, but inherits the Chapter 7 biases (position, verbosity, self-preference) and can drift. It must be anchored to a small human-labeled set and monitored (judge–human agreement over time).

See it

Three tasks. Predict the right grading approach for each — and notice which tempting choice would break.

Now explain the rule you applied: route by whether a reference exists and whether correctness is checkable. One right answer → verifier/exact match; many right answers with a reference → similarity or multi-reference; no reference → an anchored judge.

How this is graded

• Technical Correctness — you place a task on the spectrum and name the matching tool, and you know what each metric actually measures (overlap vs similarity vs rubric).
• Trade-off Awareness — you slide toward cheap/exact when you can and only pay for a judge when the task is genuinely open.
• Evaluation Rigor — you don’t grade open output with exact match, you anchor and monitor any reference-free judge, and you reach for verifiers when correctness is checkable.
• Communication — “there’s one right answer, so I’d run a verifier, not an LLM judge” is the crisp, senior call.

Industry variation

• Code / data — verifiers (tests, query execution) dominate; judges are reserved for style.
• Content / chat — mostly reference-free; the work is rubric design and judge anchoring.
• Translation / localization — multi-reference and learned metrics (e.g. COMET-style) over BLEU; human evaluation for the tail.

Stretch: the judge that grades itself

Your only option is a reference-free LLM judge, and your generator and judge are from the same model family. What’s the specific risk, and how do you keep the eval honest? (Hint: self-preference bias — the judge favors its own family’s style, so your model looks better than it is. Use a different judge model, anchor to human labels, and report judge–human agreement; if you must use the same family, at least length-control and swap-aggregate, per Chapter 7. This is exactly why a reference-free metric is never fully self-validating.)