LLM-as-judge and its biases

”GPT says it’s 95% good”

You ship a feature and evaluate it by asking a strong model to grade your model’s answers on a 1–5 rubric. It reports a mean of 4.6. In the interview, the follow-up is immediate: “Where could that number be wrong?”

Pause and list the ways an LLM judge could be systematically — not randomly — wrong, before the model’s actual answer quality even enters.

How a pairwise judge works

The common setup is pairwise: show the judge a question and two answers, ask which is better. In mini_eval we strip it to the mechanism — a deterministic mock judge, so the biases are reproducible without a live model or an API bill:

from mini_eval import Response, PairwiseJudge, position_flip_rate

a = Response("A", "Use a cross-encoder reranker.", quality=0.78)  # better, shorter
b = Response("B", "It depends; many factors weigh in here ...", quality=0.70)  # worse, longer

judge = PairwiseJudge(position_bias=0.06)   # a mild preference for whoever is first
judge.judge(a, b)            # 'A'  (A is better AND first)
judge.judge(b, a)            # may flip to 'B' if the bias exceeds the quality gap
judge.judge_debiased(a, b)   # 'A', or 'tie' when the order decided it

position_bias adds a bonus to whichever response is shown first; verbosity_bias adds a bonus per character. Real judges don’t expose these knobs — but they behave as if they had them, which is the point: if you can reproduce the behavior, you can measure and mitigate it.

| Bias | What it is | How to provoke it | Mitigation | | --- | --- | --- | --- | | Position | prefers the first (or last) answer | swap A/B order → verdict flips | judge both orders, only count order-independent wins | | Verbosity | prefers the longer answer | reward length → worse-but-longer wins | length-control the rubric; penalize unsupported length | | Self-preference | prefers its own style | judge ≈ generator family | use a different judge model; anchor to human labels |

Watch it flip

In every pair below, A is the better answer but is shorter. Predict: will the naive judge’s verdict depend on the order you present the answers in? Then toggle the order — and toggle the debias.

What you should see: the close pair’s verdict flips when you swap order — the judge is reporting position, not quality. The diagnostic is mechanical: swap the order on a sample of pairs and measure the flip rate (mini_eval.position_flip_rate); a well-behaved judge sits near zero. The debias toggle runs both orders and only declares a winner when they agree — turning the silent wrong answer into an honest “tie” you can route to a human.

How this is graded

• Technical Correctness — you name specific biases and the mechanism, not “LLMs can be biased”.
• Trade-off Awareness — you know LLM-as-judge buys cheap, scalable grading at the cost of these biases, and when a smaller human-labeled set is worth more.
• Evaluation Rigor — you don’t stop at “use a judge”; you give the flip-rate diagnostic, the swap-and-aggregate mitigation, and a monitoring plan (judge–human agreement over time). This is the exact gap the rubric weights most: naming the tool but missing the bias-mitigation is the classic miss.
• Communication — a crisp protocol (“swap order, require agreement, anchor to 50 human labels, alert if agreement drops below 0.8”) beats a paragraph of caveats.

Industry variation

• Customer-facing — judge for faithfulness to retrieved context first; verbosity bias is dangerous because fluent length reads as trust.
• Code — prefer an executable check (does it pass tests?) over a judge where you can; reserve the judge for style/readability.
• Safety — a judge for refusal/harm needs a human-anchored set and adversarial review; never let an unaudited judge gate a safety metric.

Stretch: the judge disagrees with humans

You build a 50-example human-labeled set and find your judge agrees with the humans only 68% of the time. Is the judge bad, are the humans inconsistent, or is the rubric ambiguous — and how would you tell them apart? (This is meta-evaluation: evaluating the evaluator. Start by measuring human–human agreement on the same set; if they disagree, the rubric, not the judge, is the problem.)