Human Evaluation: Building a Trace-Viewing Tool People Actually Use

An automated judge can't watch itself forever. Every judge covered so far - calibrated against human labels, tested for position and length bias - still needs a live check running underneath it, because a rubric that was accurate last quarter can quietly stop matching this quarter's traffic. Human evaluation is that last layer: not a review of everything, but a small, sustained sample read by a person, through a tool good enough that the sample actually keeps happening.

Why a Judge Still Needs a Human Layer Underneath It

Calibration is a snapshot, not a guarantee. The 100-200 traces used to calibrate a judge were a sample of the traffic at one point in time - once real traffic shifts, once a new feature ships, once users start phrasing requests differently, that snapshot stops being representative and nobody finds out unless someone keeps checking. A human layer running underneath the judge isn't there to replace it or to second-guess every verdict; it's there to keep confirming the judge is still tracking reality, the same way a company keeps auditing its own books even after the first audit came back clean.

The Real Bottleneck Isn't Judgment, It's Friction

The skill of reading a trace and deciding pass or fail isn't the hard part - a domain expert who already did error analysis can make that call in seconds. What actually kills a human-review program is the tool around that judgment: if pulling up one trace means copying an ID between three different systems, reconstructing tool calls by hand, and writing a summary nobody reads, the review doesn't get skipped through a decision to stop - it gets skipped because the friction quietly makes it not worth doing that day, and then the next day, and it never restarts. The fix isn't hiring more reviewers or lowering the bar for what counts as a review; it's removing the friction between "I want to check this trace" and "I have an answer."

What a Low-Friction Trace Viewer Actually Needs

A working trace viewer has a short, concrete list of requirements, not a long feature wishlist. It needs to render the full transcript, every tool call, and the final system state in one screen - no tab-switching between a log dashboard, a database console, and a support ticket to reconstruct what happened. It needs the same rubric used in error analysis and judge calibration sitting right next to the transcript, not living in a separate document the reviewer has to remember. And it needs a fast way to record a verdict - a keyboard shortcut or a single click, not a form with a dozen optional fields - because every extra field is one more reason a reviewer stops halfway through the sample.

Sizing the Spot-Check Cadence to Actual Risk

Not every feature needs the same review frequency, and treating them all the same either wastes reviewer time or leaves a high-risk feature under-watched. A feature where a wrong output is genuinely costly - money moved, a medical or legal claim, an irreversible action - deserves daily or near-daily sampling, even at a small sample size. A feature where a wrong output is annoying but recoverable can run on a weekly cadence with a larger batch reviewed at once. And low-stakes, low-volume features can run on a monthly or triggered-only cadence, checked mainly when something else changes. The cadence isn't a fixed policy - it should scale with the cost of a miss, the same logic that decides how much to invest in any other layer of the eval stack.

Sample-Based Review vs. Cluster Monitoring

Two different review patterns catch two different kinds of problems, and neither replaces the other. Sample-based review takes a small random slice of daily traffic and reads it in depth - this is what catches the slow, general drift in judge accuracy, the kind of miss that doesn't cluster around any one scenario. Cluster monitoring groups traffic by scenario or feature area and watches for a spike in failure rate within one specific cluster - this is what catches a new failure mode tied to a particular use case, a prompt template, or a recently shipped feature, which a small random sample might not happen to include often enough to notice. A mature review program runs both: sample-based review as the steady baseline, cluster monitoring as the early-warning system for anything narrow enough to hide inside an average.

When to Trigger a Review Outside the Normal Cadence

Some events should pull a review forward regardless of the regular schedule. A model swap behind the judge, a rubric change, a new prompt version, or a noticeable shift in the traffic mix are all reasons to run an out-of-cycle review pass before trusting the next batch of automated scores - the same logic that says a judge needs recalibration after a model change applies here: don't wait for the next scheduled check if something upstream of the review just moved.

Common Pitfalls in Human Evaluation

A handful of failure patterns show up repeatedly. Building a review tool as an afterthought - a spreadsheet, a shared doc, a handful of SQL queries - guarantees the friction problem from the start, because nobody designed for the reviewer's actual workflow. Running the same review cadence for every feature regardless of risk wastes attention on low-stakes traffic while under-watching the traffic that actually matters. Skipping the shared rubric and letting reviewers freelance produces labels that can't be compared or aggregated later. And treating human evaluation as a one-time launch checklist item instead of a standing practice lets the whole layer quietly die the moment nobody's watching to notice it stopped.

Put together with error analysis, code-based evals, and LLM-as-judge, human evaluation closes the loop on the eval stack - not by reviewing everything, but by keeping a small, sustained human check running underneath the automated layers, the one thing standing between "the judge said it's fine" and actually knowing that's still true.