LLM-as-Judge: Writing the Rubric and Calibrating the Judge Against Humans

Code-based checks catch anything with a clear right or wrong shape - a missing field, a broken schema, a state that never got written to the database. But most of what actually makes an AI product good or bad doesn't have a shape you can assert on: was the tone right, did the reply stay on brand, was the explanation actually helpful instead of just technically correct. That's the layer an LLM-as-judge fills - a model prompted with the same rubric a human reviewer would use, grading at a scale no human team can match.

Why You Need a Third Layer After Code-Based Checks

A hybrid pipeline routes the easy calls to the cheapest reliable layer and escalates the rest. Code-based checks handle the mechanical pass/fail - schema validity, required fields, whether a tool call actually changed the state it claims to have changed. What's left after that filter is the genuinely subjective slice: outputs where two competent humans could reasonably disagree about tone, completeness, or judgment. Sending that slice straight to a human reviewer doesn't scale past a handful of traces a day. Sending it to an uncalibrated model is just an automated guess dressed up as a metric. An LLM-as-judge, calibrated against real human labels, is the layer built for exactly this gap - fast and cheap enough to run on every escalated trace, accurate enough to trust because its verdicts have actually been checked against people.

Single-Criteria Judge or Pairwise Comparison, Which One to Pick

There are two ways to ask a judge model to grade something, and picking the wrong one for the question produces noisy results no amount of rubric tweaking fixes. A single-criteria judge scores one output against a fixed rubric - does this reply meet the bar, yes or no, or on a defined scale. It's the right tool when there's an absolute bar to clear: does this response violate the refund policy, does it contain a factual error, does it meet a tone standard. A pairwise comparison shows the judge two outputs side by side and asks which one is better - it's the right tool when the question is relative rather than absolute, like comparing two prompt versions or two model checkpoints on the same input. Pairwise comparison also tends to be more consistent than single-criteria scoring, because "which of these two is better" is an easier judgment than "is this a 3 or a 4 on an absolute scale" - but it only answers a relative question, not an absolute one, so it's the wrong tool when the real question is "does this meet our bar" rather than "which one is better."

Writing a Rubric That Doesn't Leave Room for the Model to Guess

A vague rubric produces a vague judge. "Rate the helpfulness of this response from 1 to 5" invites the same inconsistency a vague rubric produces in a human reviewer - two runs of the same judge model can land on different scores for the same input, because the criteria never specified what separates a 3 from a 4. A tight rubric spells out the concrete, checkable conditions that separate a pass from a fail, the same way a good error-analysis label does: not "was this a good response" but "did the response cite the correct refund policy, acknowledge the customer's stated frustration, and avoid promising a timeline the team can't guarantee." Binary criteria beat rating scales here for the same reason they beat rating scales in human labeling - a scale invites the judge to hedge in the middle instead of committing to a clear call. The rubric should also spell out edge cases the judge is likely to encounter, since a model given an ambiguous rubric doesn't ask for clarification, it just picks an interpretation and grades confidently against it.

Calibrating the Judge Against Real Human Labels

An uncalibrated judge is a model's opinion, not a metric - it has to earn trust the same way a new human reviewer would, by having its calls checked. The calibration step is mechanical: take 100-200 traces the judge has already scored, have a human independently label the same traces using the same rubric, then measure the agreement rate between the two. If the judge agrees with the human reviewer on 85-90% of cases and the disagreements are evenly spread rather than clustered around one specific failure mode, that's a judge worth trusting for the remaining traffic. If agreement is lower, or the misses cluster around a specific pattern - the judge consistently missing a subtle policy violation, say - that's a sign the rubric needs another pass before the judge earns production traffic. This step isn't optional overhead; skipping it just means the "automated eval" number the team is trusting was never actually checked against anything real.

Three Biases Worth Testing For Before You Trust a Judge

Even a judge that clears calibration can carry systematic biases that calibration alone won't surface, because the calibration sample might not happen to trigger them. The most common is position bias: in a pairwise comparison, the judge tends to favor whichever answer it sees first, independent of content - the fix is running every pair both ways and only trusting a verdict that holds under both orders. The second is length bias: judges tend to reward longer, more elaborate answers even when a shorter one is equally or more correct, which quietly punishes concision unless the rubric explicitly penalizes padding. A less universal but still common pattern is self-preference bias - a judge built on the same model family as the system being graded can show a mild preference for outputs that resemble its own writing style, which matters most when the same model (or a close relative) is both generating and judging. None of these show up as an error the judge reports - they show up as a subtle shift in what gets scored well, which is why direct bias tests, not just calibration, belong in the process before a judge is trusted at scale.

Keeping a Rubric Stable When the Grading Model Changes

Swapping the model behind the judge - a version upgrade, a cheaper model for cost reasons, a different provider - can shift scores on the exact same rubric and the exact same traces, because different models interpret the same instructions slightly differently even when the instructions haven't changed. Treating a model swap on the judge side the same way a model swap on the product side would be treated is the safe default: rerun the full calibration pass against fresh human labels before trusting the new judge's scores as comparable to the old ones. Skipping this step means a metric that looks like it moved because quality changed might have actually moved because the judge changed underneath it - and a team that doesn't know which one happened will draw the wrong conclusion either way.

Signs a Judge Is Drifting, and When to Recalibrate

A judge that passed calibration once doesn't stay calibrated forever - the traffic distribution shifts, new failure modes appear that the original rubric never anticipated, and a judge tuned for last quarter's traffic can quietly start missing this quarter's edge cases. The signal to watch for isn't a single dramatic miss - it's a slow drift in the agreement rate against a periodic human spot-check, or a pattern where the judge's pass rate keeps climbing on a rubric that hasn't gotten any looser. Treat the calibration check as a recurring practice, not a one-time gate passed at launch - a reasonable cadence is a small human recalibration sample on some regular schedule, plus an ad hoc recheck any time the product surface or the traffic mix changes meaningfully.

Common Pitfalls When Using LLM-as-Judge

A handful of mistakes show up repeatedly. Trusting a judge from day one without ever running the calibration step against human labels is the most common - it turns "automated eval" into an untested assumption. Using a vague, high-level rubric ("rate the quality") instead of concrete, checkable criteria produces noisy scores that look precise but aren't. Never testing for position or length bias leaves a systematic error baked into every score the judge produces. And treating the judge as a one-time setup instead of a living system - never rechecking calibration after a model swap or a shift in traffic - lets a judge that was once trustworthy quietly become one that isn't, with nobody noticing until the gap between the eval score and real product quality gets wide enough to be obvious.

Put together with error analysis and code-based evals, a calibrated LLM-as-judge completes the middle of the eval stack - the layer that lets a team grade the genuinely subjective slice of outputs at a scale no human team could match, without pretending a judge's opinion is a fact just because a model produced it.