Part 1 Chapter 4 Last verified 2026-06-08

Calibration and reliability

A model can rank risk perfectly and still lie about probabilities — saying 90% on a batch it gets right 60% of the time. Discrimination vs calibration, reliability diagrams, ECE and Brier built in mini_eval, and why a calibrated probability is the product when a human acts on it.

On this page
  1. AUC 0.95 — ship it?
  2. Two different questions /* anchor: cal-rank-vs-calibrate */
  3. Measuring the gap /* anchor: cal-ece-brier */
  4. See it
  5. How this is graded
  6. Industry variation
  7. Stretch: calibration doesn’t survive a shift

Calibration and reliability

AUC 0.95 — ship it?

Back to Chapter 1’s insurance model: it outputs the probability a policy files a claim, and that probability is multiplied straight into the premium. The notebook reports AUC 0.95 — it ranks risky policies above safe ones almost perfectly. A PM asks the now-familiar question: ship it?

Predict before reading on: what could be badly wrong with this model that an AUC of 0.95 would never reveal?

Two different questions

A probabilistic classifier is judged on two independent axes:

  • Discrimination — can it order outcomes? (AUC, average precision, Chapter 2.)
  • Calibration — when it says p, does the event happen a p fraction of the time? (Reliability, ECE, Brier — this chapter.)

They come apart constantly. A perfectly-calibrated coin-flipper (always says 0.5, right half the time) has zero discrimination. A perfect ranker run through a bad temperature is overconfident but still ranks flawlessly. Which axis matters depends on how the output is used (Chapter 1): a queue that only needs ordering can ignore calibration; a price, a risk a doctor acts on, or a confidence used to route (auto-resolve vs escalate) lives or dies by it.

Measuring the gap

Bin predictions by their stated probability; in each bin compare the mean confidence to the empirical accuracy (how often the event actually happened). Plotted, that’s a reliability diagram; the diagonal is perfect calibration. Two summaries, both in mini_eval:

from mini_eval import reliability_curve, expected_calibration_error, brier_score

reliability_curve(y_true, y_prob, n_bins=10)   # per-bin confidence vs accuracy
expected_calibration_error(y_true, y_prob)     # weighted mean gap (one number)
brier_score(y_true, y_prob)                     # mean squared error of the probs

Expected calibration error is the bin-count-weighted average gap, and Brier score is just mean squared error on the probabilities:

ECE=∑bnbN ∣ acc(b)−conf(b) ∣,BS=1N∑i=1N (pi−yi)2.\text{ECE} = \sum_{b} \frac{n_b}{N}\,\big|\,\text{acc}(b) - \text{conf}(b)\,\big|, \qquad \text{BS} = \frac{1}{N}\sum_{i=1}^{N}\,(p_i - y_i)^2.

See it

Two models on the same outcomes: one calibrated, one overconfident (same ranking, sharpened probabilities). Predict: for the overconfident model, will its 0.9-bin sit on the diagonal, above it, or below?

Reliability diagram2000 predictions
predicted confidence →empirical accuracy →
ECE
0.016
Brier
0.165

On the dashed line = perfectly calibrated. Below it at the right = overconfident (says 0.9, is right less often). Dot size = how many predictions in that bin.

Now explain the picture. The calibrated model’s bins hug the diagonal (ECE near 0.02); the overconfident model bows below the line at high confidence — when it says 0.9 the event happens far less often — and its ECE is several times worse, even though both models rank identically and have nearly the same Brier score. Brier blends calibration and discrimination; ECE isolates the calibration gap, which is why you report both.

How this is graded

  • Technical Correctness — you separate discrimination from calibration and know AUC says nothing about the latter; you can define ECE and Brier.
  • Trade-off Awareness — you tie which axis matters to how the output is used, and you know calibration is a post-hoc fit, not a reason to retrain.
  • Evaluation Rigor — you reach for a reliability diagram + ECE, not a single accuracy number, when a probability drives a decision; you check calibration on held-out, not training, data.
  • Communication — “it ranks fine (AUC 0.95) but it’s overconfident — ECE 0.10, so 0.9 means ~0.7; I’d temperature-scale before we price on it” is the answer.

Industry variation

  • Insurance / lending — the probability is the price; calibration and fairness are first-class, often audited, metrics.
  • Healthcare — a risk a clinician acts on must be calibrated and recall-heavy; an overconfident screen is dangerous in both directions.
  • Routing / agents — a confidence used to auto-resolve vs escalate must be calibrated, or the escalation rate drifts silently (cf. the Chapter 1 stretch).

Stretch: calibration doesn’t survive a shift

You temperature-scale your model in January and it’s beautifully calibrated. By March the user mix has changed and ECE has crept back up, even though you changed nothing. Why does calibration decay under distribution shift, and what would you put in production to catch it before a PM does? (This is calibration drift — the fix that worked on January’s distribution is wrong for March’s. The production answer is to monitor ECE on a rolling labeled sample and re-fit the temperature on a schedule — evaluation as a continuous process, the subject of Chapter 11.)