Agent Evals: Grading Outcomes, Not the Path an Agent Took

Every eval covered so far in this pillar - unit tests, an LLM judge, a human reviewer - grades a single output: one summary, one reply, one classification. An agent breaks that assumption immediately. It reads data, calls tools, decides what to do next, and does this across many turns before producing anything that looks like a final answer. Grading that chain the way you'd grade one output misses almost everything that actually went wrong - or worse, flags a correct run as broken because it didn't match the script the grader expected.

Why Grading an Agent Means Grading a Chain, Not a Single Answer

A single-turn eval has one thing to check: the output. An agent eval has to check the transcript - every tool call, every intermediate decision - and, just as important, the actual state the agent left behind in the world once it stopped. A booking system, a file, a database row, an email that was or wasn't sent: that state is the closest thing to ground truth an agent eval has, and a grader that only reads the agent's final message is grading marketing copy, not the work.

What "Outcome" Actually Means: The State Left Behind, Not the Transcript

Outcome-based grading means checking the real, external state a task was supposed to change, not the words the agent used to describe what it did. A task like "book the room" has a clean, checkable outcome: query the calendar system directly and see if a matching booking exists. Wherever a task produces this kind of checkable side effect, a code-based outcome check is the strongest grader available - cheap, deterministic, and immune to an agent that talks a good game but didn't actually do the work. Not every task has a state this clean; "explain the refund policy clearly" still needs the LLM-as-judge or human layers from earlier in this pillar. Outcome-state checking doesn't replace those - it adds a ground-truth layer for the tasks that have one.

Why the Grader Can't Demand One Fixed Sequence of Steps

Agents routinely solve the same task through different, equally valid tool orderings - check inventory before confirming price, or the reverse; read the existing record before writing, or write and reconcile after. A grader that requires one exact sequence punishes a correct run for taking a different route, and a team that watches its eval numbers swing on path changes that have nothing to do with actual quality stops trusting the eval entirely. The fix isn't to abandon structure - it's to check the outcome state as the primary signal, and layer a small set of hard "must never happen" assertions on top (never call a delete action, never skip a required confirmation step) rather than pinning down the entire path.

Non-Determinism: Why One Trial Never Tells You Enough

The same agent, given the identical input, can succeed on one run and fail on the next - sampling variance, tool timing, a retry that resolves differently. A single trial per task produces a number that looks precise but is actually noise wearing the costume of a result. Running multiple trials and reporting pass@k (the agent gets credit if at least one of k trials succeeds) measures a capability ceiling - what the agent can do when it gets lucky. Reporting a consistency rate (the agent must succeed in all k trials) measures what you can actually ship - what the agent does reliably, every time. These answer different questions, and picking the wrong one gives a false sense of security: a task with a high pass@5 but a low consistency rate looks solved until it's shipped and fails a third of the time in production.

Where the Eval Budget Actually Goes

Most teams budget for the grading step and forget that running the agent itself, multiple times per task, across every task in the suite, is usually the largest cost line - each trial is a full multi-step, multi-call agent run, not one cheap model call. Budgeting only for the judge or the human review layer while ignoring the compute cost of the trials that produce the transcripts in the first place is how an eval budget blows past its estimate by an order of magnitude.

If a Frontier Agent Fails Almost Every Trial, Look at the Task First

A well-specified task that a competent person could complete should rarely fail on every single trial for a frontier agent. When that happens, the task or the grader is the more likely culprit, not the agent's raw capability. Check three things before blaming the agent: is the task description ambiguous, or does it assume information the agent has no way to obtain; does the grader have a bug, like checking the wrong field or the wrong final state; is the tool or environment itself broken, returning stale or malformed data that no agent could work around. A near-100% failure rate is a signal to audit the eval, not a verdict on the agent.

Eval Suite Saturation: When the Benchmark Stops Discriminating

A suite saturates when most of the agents being compared score near the ceiling - or near the floor - and the numbers stop being able to tell a genuinely better agent from a worse one. The tell is a score delta between model versions that's shrunk down to noise level, while real-world quality clearly still differs between them; teams keep reporting the same benchmark release after release even though it stopped moving in any meaningful way months ago. The fix mirrors the discipline from error analysis: retire or raise the difficulty of tasks that have saturated, and add new tasks sourced from actual production failures rather than continuing to run an old suite just because it's already built.

Common Pitfalls in Agent Evals

A handful of failure patterns recur across agent eval setups. Grading the transcript's wording instead of the outcome state lets a hallucinated success pass as real. Requiring one fixed path punishes correct answers that took a different route and erodes trust in the eval numbers. Trusting a single trial treats noise as signal and hides exactly the kind of instability that ships to production. Letting agent compute cost balloon unnoticed happens when nobody budgets for the multiplication of trials times tasks. And running the same saturated suite release after release gives a team a number that looks stable while telling them nothing real about whether the agent actually improved.

Put together with error analysis, code-based evals, LLM-as-judge, and human evaluation, agent evals complete the five pieces of an eval system - the one layer that has to grade a chain of decisions and the state they leave behind, instead of a single, easily-checked answer.