Confidence intervals and statistical rigor

Two points better — ship it?

Model A scores 0.59 F1 on your 600-example eval set. You change the prompt, rerun, and model B scores 0.65. Six points! In the interview you’re half-imagining, the lead engineer asks: “Nice — roll it out?”

Before you answer: how many of those six points would survive if you’d drawn a different 600 examples? Write down a yes/no — would you ship — and one number you’d want to see first.

A metric is a sample, so give it error bars

You usually can’t collect a fresh eval set on demand — but you can simulate drawing new ones from the set you have. The bootstrap: resample your examples with replacement to the same size, recompute the metric, and repeat a thousand times. The spread of those thousand numbers estimates the metric’s sampling distribution; its middle 95% is your confidence interval.

In mini_eval it’s a few lines on top of the Chapter 2 metrics — resample, recompute, take percentiles:

from mini_eval import recall, bootstrap_ci

lo, hi = bootstrap_ci(y_true, y_score, threshold=0.5,
                      metric_fn=recall, n_boot=1000, ci=0.95)
# recall = 0.80  (95% CI 0.71–0.88)

Formally, each resample gives an estimate $\widehat{\theta}^{*(b)}$, and the percentile interval is just the empirical quantiles of those $B$ values:

The width is the whole point. It shrinks roughly with $1/\sqrt{n}$: quadruple the eval set, halve the interval. A recall of “0.80” from 50 examples and “0.80” from 5,000 are not the same claim — and the CI is what makes the difference visible.

Is the difference real?

Back to B vs A. The right comparison is paired: both models ran on the same examples, so a hard batch drags both scores down together. That shared difficulty is often the biggest source of variance, and pairing cancels it — compare within-example, not set-against-set. Bootstrap the difference and read whether its interval clears zero:

from mini_eval import f1, paired_diff_ci, permutation_test

res = paired_diff_ci(y_true, score_a, score_b, threshold=0.5, metric_fn=f1)
res["diff"], (res["lo"], res["hi"]), res["excludes_zero"]
# +0.06, (0.00, 0.12), True  — but only just

p = permutation_test(y_true, score_a, score_b, threshold=0.5, metric_fn=f1)
# p ≈ 0.05 — shuffle which model owns each example; how often does chance match the gap?

Two views of one question. The effect-size CI says how big the difference plausibly is; the permutation test says how surprising it would be under “no real difference.” You want both — a p-value with no effect size hides whether a “significant” win is trivially small.

See it

The left tab is the bootstrap CI shrinking as the eval set grows; the right tab is B-vs-A. Predict first: on the right tab, B leads A by six F1 points on 600 examples — do you expect the 95% CI on the difference to clear zero?

Now explain what you saw. The difference is barely significant — the CI’s lower edge sits just above zero and the permutation p is right at 0.05. That’s the honest answer to “ship it?”: the lead is probably real, but one unlucky resample would erase it, so you’d want a guardrail (Chapter 11’s online check) or more data before betting the roadmap on it.

How this is graded

• Technical Correctness — you treat a metric as an estimate with a sampling distribution, and can describe the bootstrap and a paired test without hand-waving.
• Trade-off Awareness — you weigh “collect more eval data” against shipping on a thin lead, and know a p-value and an effect size answer different questions.
• Evaluation Rigor — the rubric’s weighted dimension here: you report a metric with a CI, compare models paired, and don’t call a 6-point lead a win until its interval clears zero. A bare point estimate is the classic miss.
• Communication — “B is up 6 F1, 95% CI [0.0, 0.12], p≈0.05 — likely real but marginal, I’d gate it” beats both “B is better” and a wall of statistics.

Industry variation

• Agentic / code — outputs are stochastic; report mean ± std over n≥5 runs before any cross-model claim, then test the difference.
• Fraud / risk — metrics ride on a few rare positives, so bootstrap CIs on small holdouts are wide; decisions wait for tighter intervals or cost-weighting.
• Research / leaderboards — many models compared at once means a single 0.05 threshold over-claims; correct for multiple comparisons (the stretch).

Stretch: you compared twenty prompts

You bootstrap twenty prompt variants against the baseline and three clear the 0.05 bar. How many “winners” would you expect from pure chance if none truly helped — and what changes about how you’d report the best one? (This is the multiple-comparisons problem: at $\alpha = 0.05$, twenty independent tests yield about one false positive on average — and a ~64% chance of at least one. A Bonferroni or Benjamini–Hochberg correction — or a held-out confirmation set — is how you keep from shipping noise.)