LLM-as-a-judge: a practical guide

Exact match and code checks only work when there's one right answer. Summaries, chat replies, support responses, and creative writing don't have one — so you need a grader that can reason about quality. An LLM judge reads the output, applies your rubric, and returns a score plus a short rationale, at a fraction of the cost of human review.

The judge is usually a stronger or larger model than the one being graded, but it doesn't have to be: consistency and calibration matter more than raw capability.

The rubric is the eval. Vague instructions like "rate the quality from 1–5" produce noisy, unrepeatable scores. Instead, break "good" into specific, checkable criteria:

• Binary checks: "Does the answer cite a source?" "Is it under 100 words?" Binary criteria are the most reproducible.
• Anchored scales: if you need 1–5, define each point ("3 = correct but misses a key constraint"). Unanchored numbers drift.
• Task-specific rules: tone for support replies, faithfulness for RAG, format for extraction. One rubric per task, not one rubric for everything.

Ask the judge to output its reasoning before the score. Chain-of-thought grading improves agreement with humans and makes failures debuggable.

Pointwise scoring asks "how good is this answer on a scale of 1–5?" It's simple and produces one number per output — good for dashboards and regression tracking.

Pairwise comparison asks "which answer is better, A or B?" Models (and humans) are far more reliable at ranking two options than at assigning absolute numbers, so pairwise is the better choice when comparing two prompts, models, or versions. Randomize the A/B order per sample and swap positions to cancel position bias.

Don't grade cold. Include 2–4 worked examples in the judge prompt: an input, an output, and the score with its rationale. These examples anchor the judge's standards far better than any description of the rubric can.

Pick examples at the boundaries — a clear pass, a clear fail, and one borderline case. Borderline examples teach the judge where your line is. Keep the examples stable across runs so scores stay comparable over time.

LLM judges have well-documented biases. Design your prompt and protocol around them:

• Length bias: judges favor longer answers. Add "do not reward length for its own sake" and check score vs. length correlation in your data.
• Position bias: in pairwise mode, the first option wins too often. Shuffle order and run both A/B and B/A.
• Self-preference: a judge favors outputs in its own family's style. Grade blind — strip model identity — and consider a judge from a different model family.
• Verbosity and confidence bias: confident, well-formatted answers score higher than equally correct plain ones. Ask for rubric criteria, not vibes.

An unvalidated judge is a broken ruler. Label 50–200 examples yourself (or with your team), run the judge on the same set, and measure agreement — percent agreement or Cohen's kappa. As a rule of thumb: below ~80% agreement, fix the rubric or the examples before trusting any score the judge produces.

Re-validate whenever you change the judge model, the rubric, or the task. Judge quality doesn't transfer automatically.

Run the judge in CI on every prompt or model change, keep the judge model version pinned, and log the judge's rationale alongside each score so failures can be reviewed. Keep a small human-review loop running — a weekly sample of graded outputs — to catch drift the moment it starts.