Production evaluation and monitoring
Offline eval is a snapshot; production is a stream. Why a frozen offline metric stays green while live quality drifts, the online signals and guardrails that catch it, drift detection, cost and latency as first-class metrics, and the alerting that turns evaluation into a continuous process.
On this page
Production evaluation and monitoring
The dashboard is green and users are angry
Your offline F1 has been flat at a healthy number for months — green dashboard, no alerts. Yet support tickets are climbing and the human-override rate is creeping up. In the interview: how can the eval say everything’s fine while the product gets worse?
Predict: what does a frozen offline eval set structurally fail to see?
Offline is necessary, not sufficient
Offline eval (everything in Chapters 2–10) answers “is this good enough to ship?” on data you control. Production adds three things it can’t provide:
- Online evaluation — quality measured on live traffic: a sampled human review, an LLM judge on real outputs (Chapter 7), implicit signals (thumbs, edits, re-asks, escalations).
- Guardrail metrics — bounds you alert on, not optimize: refusal rate, override rate, p95 latency, cost per request, safety-flag rate. They catch regressions a quality average hides.
- Continuous comparison — canary / shadow / A-B in production, so a new version is judged against the old on current traffic, with the statistics of Chapter 3.
Drift is what offline misses
The core production failure is distribution shift: the input mix moves away from your eval set, so the model degrades on reality while scoring fine on the frozen set. What you monitor instead:
- Input drift — embedding/feature distribution of live inputs vs the eval set.
- Output-quality drift — a rolling judged or human-labeled sample of live outputs; calibration drift (ECE creeping up, Chapter 4); judge–human agreement over time.
- Business / guardrail signals — override rate, escalation rate, conversion — the things that move before anyone files a bug.
See it
A frozen offline metric and a live online signal over 30 days, with a guardrail. Predict: will the offline line move when the input distribution shifts on ~day 12?
The offline metric (frozen eval set) is flat and green all month. The live signal drifts as the input mix changes and breaches the guardrail on day 22 — a failure the offline dashboard never shows.
Now explain the divergence. The offline line is flat — it can’t move, the set is fixed — while the online signal drifts past the guardrail and trips an alert on day 22. The lesson isn’t “offline eval is bad”; it’s that offline eval and a live guardrail catch different failures, and you need both.
Cost and latency are eval, too
A correct answer that costs $2 and takes 30 seconds can be a worse product than a slightly-worse answer that costs 2¢ and returns instantly. In production, evaluation includes:
- Latency (p50/p95/p99), throughput, cost per request / per success.
- SLOs with error budgets: bound the operational metrics and alert on the budget.
- Quality-per-dollar trade-offs across model tiers — often the real deploy decision, not raw quality.
How this is graded
- Technical Correctness — you distinguish offline from online eval and name concrete online signals and guardrails, not “we monitor it”.
- Trade-off Awareness — you treat cost/latency as eval and reason about quality-per-dollar across model tiers.
- Evaluation Rigor — the weighted dimension: you’d monitor drift, keep a live judged sample, alert on guardrails, and compare versions with production statistics.
- Communication — “offline gates launch; a live override-rate guardrail and a canary catch the drift offline can’t see” is the answer that signals production experience.
Industry variation
- High-traffic consumer — implicit signals and A-B dominate; latency/cost SLOs are hard gates.
- Regulated (finance/health) — drift and calibration monitoring are compliance requirements; human-in-the-loop sampling is mandated.
- Low-traffic / high-stakes — not enough volume for A-B; lean on human review of a large fraction and tight guardrails.
Stretch: your monitor needs labels
Online quality monitoring needs to know if live answers were good — but production has no ground truth. Where do the labels come from, fast and cheap enough to trend daily? (Hints: an LLM judge anchored to a small human set (Chapter 7); implicit labels from user behavior — edits, thumbs, re-asks, escalations; delayed ground truth that arrives later (did the flagged transaction charge back?); and a small human-reviewed sample as the calibration anchor for all of the above. Each is noisy — the skill is triangulating several weak signals into one you can trust.)