Code-Based Evals: Write Assertions Only for What a Machine Can Verify

Once error analysis has surfaced a failure taxonomy, the next question is how to write an evaluator for each category. For most categories, the cheapest answer is a few lines of code, not a model call. But "cheapest" doesn't mean "works for everything" - an assertion only holds when the thing being measured is genuinely mechanically checkable, and knowing where that boundary sits matters more than knowing how to write an assertion.

Code-based eval is the cheapest, fastest, and most deterministic layer of an eval suite. It's also easy to misuse in two opposite directions: skipped because it looks "too simple" next to LLM-as-judge, or forced onto things it can't actually measure, producing brittle assertions and a steady stream of false regressions.

Why Code-Based Eval Is the Cheapest Layer, Not the Only One

A code assertion runs in milliseconds, costs nothing per call, and returns an absolutely deterministic result - the same input always produces the same pass/fail. That's why it should always be the first layer in any eval pipeline: cheap enough to run on every trace, every deploy, with no budget tradeoff to weigh. But the price of that determinism is narrow scope - an assertion only answers questions that already have a rule-based right or wrong, not questions that require judging context. Confusing "cheapest" with "sufficient for everything" is the fastest way to end up with a suite that both misses real bugs and fires constant false alarms.

From Failure Taxonomy to a Concrete Assertion

Every category in the taxonomy from error analysis deserves one question before a single line of code gets written: does this category have a mechanical rule for telling right from wrong? A category like "schema drift on edge input" is almost always mechanically checkable - there's a correct data shape, and everything else is wrong. A category like "tone mismatch" is not - no code rule reliably defines "sounds cold" as a binary fact. Answering this first decides whether a category belongs in the code-based layer or needs to escalate somewhere else, and doing it before writing the assertion avoids forcing a subjective judgment call into an if/else statement.

Tight Assertions and Rigid Ones, Where the Line Actually Sits

The line comes down to what the assertion is actually matching: a real invariant (the total is correct, a required field is present, the final state matches the request) or a surface form (exact wording, sentence order, phrasing). Asserting on the invariant keeps the necessary rigor while still accepting many valid ways of phrasing a correct answer - which is exactly what a good unit test does for traditional code, except an LLM's valid output space is far wider than a pure function's. Asserting on surface form is tight in the wrong place: it reliably catches every time the wording changes, and reliably misses every time the output is substantively wrong but happens to match the old pattern.

Check the Final State, Not Just What the Agent Claims

For an agent that calls tools and leaves behind real state in a system, the strongest assertion doesn't read the transcript - it reads that state back. An agent can narrate an action in flawless, accurate-sounding language while the underlying action actually failed or wrote the wrong data, and an assertion that only parses the reply text will never catch that gap. Querying the database, the tool-call log, or any source of truth independent of the agent's own narration is the only way an assertion measures what the user actually got, not what the agent said it did.

The Three Assertion Types You'll Actually Reach For

Most assertions in an eval suite fall into one of three shapes: schema/required-field checks, real system-state checks, and exact-string matches. The first two are almost always safe because they verify an objective fact - the structure is correct or it isn't, the data row exists correctly or it doesn't. The third is the only shape that carries inherent risk: it implicitly assumes there's exactly one correct way to phrase an answer, an assumption that's almost always false for natural language. Recognizing which shape an assertion is the moment you write it heads off most of the false regressions that would otherwise surface weeks later.

Turn a Trace You've Already Read into a Fixed Test Case

Every failing trace surfaced through error analysis is a free opportunity to build a regression test case grounded in fact, not assumption: take that exact input, that exact wrong output, write an assertion that expresses exactly why it was wrong, and add it to the suite that runs on every prompt or model change. This is far cheaper than trying to imagine every possible test case up front, because each one traces back to a failure that actually happened rather than one someone guessed might happen. Over time, this growing set of test cases becomes the cheapest defense against fixing one bug by accidentally reviving another that was already fixed.

The Per-Run Cost, and Why You Should Push as Much Down to This Layer as Possible

The cost gap between a code assertion and one LLM-as-judge call isn't a few percentage points - it's several orders of magnitude, because one side only costs compute and the other costs a full inference pass. The practical consequence: every failure category successfully converted from "needs a judge" to "measurable by assertion" doesn't just save money, it frees up judge budget for the cases that genuinely require subjective judgment. A healthy eval pipeline always pushes as many traces as possible through the code-based layer first, escalating to the more expensive layer only when there's genuinely no other way.

The Signal That Means You Need to Escalate to LLM-as-Judge

There's one recurring signal more reliable than any theoretical rule: if an assertion keeps growing another "or X, or Y, or Z" clause to cover every valid phrasing, what's being measured has stopped being an invariant and quietly turned into a quality judgment. Another signal is when two engineers read the same failing assertion and disagree about whether the output was actually wrong - that disagreement means the code rule is no longer sufficient to define right and wrong, and what's needed is a judge capable of weighing context instead of a rigid if statement.

Common Pitfalls in Writing Code-Based Evals

A few mistakes show up in most teams: writing an assertion that matches one sample run's exact output, turning the suite into a list that only accepts one specific phrasing; checking internal implementation details (variable names, the order tools got called) instead of the final outcome the user actually sees; letting assertions go stale when a prompt or model changes, so the suite quietly starts grading a deliberately changed behavior as a failure; and failing to version test cases alongside prompt changes, so no one can tell whether an assertion is protecting the old behavior or the new one. Keeping assertions anchored to the invariant, not the surface form, is the only way this cheapest layer of the eval suite stays trustworthy across every model change that follows.