Data integrity: leakage, contamination, reproducibility
The fastest way to a great eval number is a broken eval set. Train/eval overlap, parametric-memory leakage in RAG, adaptive overfitting from tuning on a fixed set, benchmark contamination, and the reproducibility hygiene — frozen sets, pinned versions, seeds — that makes a number mean something.
On this page
Data integrity: leakage, contamination, reproducibility
The 0.98 that means nothing
A teammate is thrilled: the new eval reports 0.98. Before you celebrate, ask the question that has saved more projects than any metric choice: where did the eval data come from, and could the model have seen it? More great-looking numbers die here than at the hands of a wrong metric.
Predict: name one way an eval set can produce a near-perfect score that says nothing about whether the model actually works.
The leakage modes
Three ways information sneaks from where it shouldn’t:
- Train/eval overlap. Eval examples — or near-duplicates — are also in training, so the score measures memorization. A random split is the usual culprit: it scatters near-duplicate records across train and test. Split by time or entity before dedup so echoes can’t straddle the boundary.
- Parametric-memory leakage (RAG). The answer is in the model’s pretraining memory, so the generator is right even when retrieval fails — masking your retrieval and faithfulness metrics (Chapter 9). Test on documents the model could not have memorized: private, post-cutoff, or synthetic.
- Adaptive overfitting. Every prompt you tune against a fixed eval set leaks a bit of that set into your system; after enough rounds the gains are overfitting, not progress. Keep a held-out confirmation set you touch rarely, and refresh the dev set.
Decontamination, concretely
Detecting overlap isn’t magic — it’s bookkeeping:
- Exact + near-duplicate matching (hashing, n-gram / MinHash overlap) between eval and training corpora.
- Canary strings planted in training data: if they surface at eval time, you have a leak path.
- Perturbation tests: paraphrase or numerically alter eval items; a score that craters under paraphrase was memorized, not understood.
- Temporal holdout: evaluate only on data created after the training cutoff (the LiveBench idea from Chapter 8), which is contamination-proof by construction.
See it
Three setups that each report a great number. Predict the leak in each before you reveal it.
Each setup looks fine and reports a great number. Pick the leak that makes the number a lie.
“We built our eval set by sampling from the same conversation logs the model was fine-tuned on. Accuracy is 0.96.”
“Our RAG eval uses Wikipedia articles that are also in the model's pretraining corpus. Answer accuracy is high.”
“We've been tuning prompts against the same 500-example eval set for three months. Our score keeps climbing.”
Now explain the through-line: in every case the model had seen the answer by some path — the same examples, its own memory, or the eval set it had been tuned against — so the score measured recall of that path, not capability.
Reproducibility hygiene
Even a clean eval set is worthless if you can’t reproduce the number. The checklist:
- Freeze the eval set and version it; never edit it silently between runs.
- Pin everything that moves: model version/snapshot, decoding params (temperature, top-p), prompt template, and library versions.
- Seed stochastic steps, and for unavoidable stochasticity (sampling, agents) report mean ± std over n runs (Chapter 3), not one run.
- Record the config with the result so “0.73” always carries which model, which prompt, which set, which day.
How this is graded
- Technical Correctness — you name the leakage modes precisely and the decontamination check for each, not just “avoid leakage”.
- Trade-off Awareness — you know a random split is convenient but dangerous, and why temporal/entity splits cost more but buy trust.
- Evaluation Rigor — the weighted dimension: you treat the data as the first thing to audit, keep a held-out confirmation set, and make runs reproducible.
- Communication — “before I trust 0.98 I’d check train/eval overlap and whether we’ve been tuning on this set” is the answer that flags the senior instinct.
Industry variation
- Finance / healthcare — temporal leakage is the cardinal sin; splits must respect time, and audited reproducibility is mandatory.
- Search / RAG — parametric-memory leakage dominates; private and post-cutoff corpora are how you isolate retrieval.
- Research — contamination disclosure and decontamination reports are now expected alongside any benchmark claim.
Stretch: you inherited the eval set
You join a team with a year-old eval set that everyone trusts and that the model has been tuned against the whole time. You suspect it’s compromised but can’t just throw it out. How do you find out — and what do you build alongside it? (Hint: build a fresh, held-out set from recent data and compare the model’s score on old vs new; a large gap is the tell. Then run overlap/decontamination on the legacy set, and move the team’s decisions onto the fresh confirmation set while keeping the old one only as a smoke test.)