Benchmark literacy
MMLU, GPQA, HumanEval, SWE-bench, GAIA, Chatbot Arena, HELM, LiveBench — what each actually measures, and the three threats that turn a headline score into a lie: saturation, contamination, and construct validity. How to read a leaderboard like an engineer, not a marketer.
On this page
Benchmark literacy
”It’s #1 on the leaderboard”
A new model tops MMLU and your feed is full of charts. A stakeholder forwards one: “This is the best model now — should we switch?” The junior answer is “yes, it’s #1.” The senior answer starts with three questions about the benchmark itself.
Predict: name one reason a chart-topping benchmark score might not mean the model is better for your product.
The landscape
A rough map of what the headline benchmarks probe:
| Benchmark | Measures | Format | Watch out for | | --- | --- | --- | --- | | MMLU | broad knowledge / exam reasoning | multiple choice | saturated; contaminated | | GPQA | hard graduate science reasoning | multiple choice | “Diamond” subset; small | | HumanEval / MBPP | function-level code | write-a-function | tiny; heavily contaminated | | SWE-bench | real repo bug-fixing | patch a GitHub issue | harder; closer to real work | | GAIA / WebArena | agentic, tool use, multi-step | task success | expensive; stochastic | | Chatbot Arena | human preference, head-to-head | Elo from votes | preference ≠ correctness | | MT-Bench | multi-turn quality | LLM-as-judge | judge biases (Chapter 7) | | HELM | holistic, many scenarios + metrics | battery | breadth over depth | | LiveBench | contamination-resistant, fresh | rotating, post-cutoff | newer; smaller community |
The point isn’t to memorize the table — it’s to ask, for any claim, which of these a score comes from and whether that’s the construct you care about. A coding agent for your monorepo cares about SWE-bench, not HumanEval; a chat product cares about Arena and your own preference data, not GPQA.
Three threats to a benchmark claim
Almost every misleading leaderboard reduces to one of three:
- Saturation. When top models cluster at 89–90%, the remaining gap is label noise and ambiguous items. A “0.3-point SOTA” near the ceiling is noise dressed as progress (Chapter 3’s CI would show the bars overlapping). Fix: move to a harder or fresher benchmark.
- Contamination. A public test set older than the training cutoff has probably leaked into pretraining, so the score measures memorization. Fix: post-cutoff or rotating sets (LiveBench), canary strings, or perturbed/paraphrased items — if the score craters under paraphrase, it was memorized.
- Construct validity. The benchmark is clean and unsaturated but measures the wrong thing for your use — HumanEval puzzles vs your production codebase, exam trivia vs your domain. Fix: pick (or build) a benchmark whose tasks resemble yours; ultimately, your own eval set (Chapter 12).
See it
Three real-sounding claims. Predict the sharpest reason to distrust each before you reveal it.
Each line is a real-sounding benchmark claim. Pick the single sharpest reason to distrust it before you act on it.
“Our model scores 95% on a popular benchmark first published in 2021. Its training data runs through 2024.”
“Three frontier models score 89–90% on MMLU; ours hits 90.3% — a new state of the art.”
“Our model tops the HumanEval coding leaderboard, so it's the best choice for our large production codebase.”
Now explain the pattern: in each case the number was real but the claim built on it wasn’t — because the benchmark was saturated, contaminated, or measuring the wrong construct. That triage (which of the three?) is the literacy.
How this is graded
- Technical Correctness — you know what the named benchmarks measure and their format, not just the acronyms.
- Trade-off Awareness — you weigh a public benchmark against your own eval and know when each is the right evidence.
- Evaluation Rigor — you reach for the three-threat checklist (saturation / contamination / construct validity) and pair any score with a CI and a relevance argument before acting on it.
- Communication — “MMLU is saturated and probably contaminated; for our codebase I trust SWE-bench-verified and a private set far more” is the senior answer.
Industry variation
- Frontier labs — contamination control and fresh/held-out benchmarks are core; saturation drives the constant push for harder evals.
- Applied teams — public benchmarks are a screen; the real decision rides on a domain eval set, because construct validity rarely holds for a niche product.
- Regulated domains — benchmarks must map to documented requirements; a generic leaderboard rarely satisfies an auditor.
Stretch: the benchmark you can trust
If every public benchmark is saturating, contaminating, or off-construct, what do you actually trust to pick a model? (The answer threads the whole guide: a private, held-out eval set built from your own tasks, scored with metrics chosen from how the output is used — Chapter 1 — with confidence intervals — Chapter 3 — and refreshed so it can’t leak. That’s the Chapter 12 capstone: designing the eval, not borrowing one.)