LLM Evals: Measuring AI Quality Before It Quietly Breaks Your Product
Shipping an LLM feature without evals means nobody can answer "is the new version actually better". Evals turn "looks fine" into a number you can measure across every prompt, model, or data change.
Most teams ship their first LLM feature the same way: eyeball a few dozen outputs, decide it looks fine, and push it live. That's the fastest way to ship, and for the first week it feels like it's working. The trouble starts once the prompt changes, the model gets upgraded, or real traffic starts arriving - and nobody on the team can answer "is the new version actually better, or does it just feel that way?"
Evals exist to answer exactly that question - by turning "looks fine" into a number you can compare across time, across prompt edits, across model swaps, across data changes.
Eval Loop High-Level Flow
Why LLMs Can't Be Tested Like Traditional Software
Traditional software tests have one correct answer: add(2, 3) must return 5, no room for interpretation. LLMs don't work that way - ask the same question twice and the model may answer differently, with both answers equally valid. "Summarize this paragraph" has dozens of acceptable summaries; "was this reply empathetic" depends on who's reading it; an output that looks polished across a handful of demo cases can still fall apart on the messy, abbreviated, or adversarial input a real user types in.
Because outputs are probabilistic rather than deterministic, an eval isn't a test you run once and file away. The probability of a correct answer shifts the moment the prompt, model, or data changes - so the eval has to rerun at every one of those changes, not just once at launch.
What an Eval Actually Is
Stripped to its parts, an eval is four things: a task (a specific input paired with success criteria), a trial (one attempt at that task - run multiple trials because output varies run to run), a grader (the logic that scores a trial), and a transcript (the full record of that trial: the output, any tool calls, intermediate reasoning steps). For a single-turn task like summarization or classification, the transcript is just the final output. For a tool-calling agent working across multiple steps, the transcript is the entire chain of actions - and that difference is the biggest gap between evaluating a single model call and evaluating an agent.
How a Change Moves Through the Eval Ladder
Cheap, fast checks run first on every change. Slower, more expensive levels only run when the change is big enough to warrant them.
"Reword the refund-policy clause in the support-reply prompt."
42/42 assertions pass in 8s. No known regression triggered.
Not required — change is scoped and fully covered by existing assertions.
Not required — ships straight to production.
The Three-Level Ladder: From Unit Tests to A/B Tests
A mature eval system doesn't run one kind of check for every change - it runs three levels, each with a different cost and a different job.
Level 1 - Unit Tests are deterministic assertions: does the JSON match the schema, does the generated code pass its tests, is a required field present. They run in seconds, cost almost nothing, and run on every commit. This is the layer that catches known regressions before they reach production.
Level 2 - Model & Human Eval kicks in once the question stops being a mechanical pass/fail and becomes "is this actually a good answer." The team logs real transcripts, a domain expert reads and labels them (binary, not a rating scale), and those labels are then used to calibrate an LLM judge that can grade at a larger scale.
Level 3 - A/B Testing should only turn on once the product is mature enough to trust Levels 1 and 2: split real traffic, measure real user behavior, and confirm that an improvement on the eval set actually translates into a better experience - not just a better-looking number on a static dataset.
Before You Write an Evaluator, Read the Traces
The most common mistake isn't a missing tool - it's writing an evaluator before understanding where the product is actually breaking. The right order runs the other way: read a few dozen real transcripts, label them pass or fail by hand, and let that reading surface which failure modes actually recur - only then write the assertion or rubric that targets them.
A few small rules make an outsized difference here. Binary labels (pass/fail) force a clear call, while a 1-5 rating scale mostly produces arguments over the line between a 3 and a 4. One person - not a committee - should own what "good" means for a given feature, so labeling doesn't drift between reviewers applying different standards. And this reading shouldn't be outsourced entirely: the loop between "seeing a real failure" and "fixing the product" is what produces improvement, and cutting that loop off from the people making decisions breaks the improvement along with it.
Which Grader Fits This Case?
The same three grader types keep showing up. Which one to reach for depends on whether correctness can be checked mechanically, and what a wrong answer costs.
"Extract order ID, SKU, and quantity from this email into JSON matching our schema."
Correctness is a yes/no question a script can answer.
Schema validation is exactly what code is good at: fast, free, and it never disagrees with itself on the same input.
Caveat: Only tells you the shape is right — it says nothing about whether the extracted values are actually correct.
If a script can check correctness, always start there — it's free and never drifts.
Three Kinds of Graders: Code, Model, and Human
Once you know what to grade, the next question is who grades it.
Code-based graders check anything with a clear right or wrong answer - a format, the presence of a field, whether a test case passes. They're cheap, fast, and never change their mind between two runs on the same input. But they only confirm the shape is right, not that the content is actually correct.
Model-based graders (LLM-as-judge) step in once the criteria turn subjective - tone, helpfulness, whether a reply stays on brand. A model prompted with the same rubric a human reviewer would use can grade thousands of outputs a day. But an uncalibrated judge is just an automated guess: before trusting it, its verdicts need to be checked against 100-200 human labels on the same data, measuring whether agreement is actually high enough.
Human graders remain the last layer for cases that are ambiguous, high-stakes, or too new to have enough data to train a judge on. Human review doesn't scale to every request - and it doesn't need to; it only needs to scale to the slice where a wrong call is genuinely expensive.
Blended Cost per Eval Run Calculation
Model scenario: a mature eval suite running 85% unit tests, 12% LLM-as-judge, 3% human review.
Catches known regressions on every PR
Scores subjective criteria at scale
Reviews ambiguous, high-stakes, or novel cases
The Economics Behind Mixing Three Grader Types
No team can afford to route 100% of eval runs through human review, and no team should fully trust an uncalibrated judge for every kind of failure either. The economics push hard toward mixing: let most eval runs pass through the cheapest layer that's still reliable enough, reserve the middle layer for scale, and spend the most expensive layer - people - only on the slice where the cost of a wrong answer is large enough to justify it.
When Agents Enter the Picture, Evals Change
Grading a single model call means grading one output. Grading an agent that calls tools, reads data, and decides its next move across multiple turns means looking at both the transcript and the final outcome - the actual state left behind in the system, like whether a booking really exists correctly in the database, not just what the agent said it did.
That difference carries two easy-to-miss consequences. First, a grader shouldn't demand the agent follow one fixed sequence of steps - grade what the agent produced, not the path it took, since multiple valid paths can lead to the same correct outcome. Second, if a frontier agent fails nearly every trial on a task, that's more often a sign the task is broken or the grader itself has a bug than proof the agent is genuinely incapable - the task deserves a second look before the agent gets the blame.
Five Pieces of a Complete Eval System
Put together, a complete eval system needs five pieces, each one its own decision worth digging into:
Error analysis. The process of reading and labeling traces before writing any evaluator, plus deciding who actually owns the call on what counts as "good."
Unit-test / code-based evals. How to write assertions tight enough to catch regressions without being so rigid they reject a correct answer that just took a different path.
LLM-as-judge. How to design the rubric, choose between single-criteria scoring and pairwise comparison, and calibrate the judge against human labels.
Human evaluation. How to build a trace-viewing tool that doesn't create friction, and what spot-check cadence makes sense once an automated judge is in place.
Agent evals - how to define outcomes, handle non-determinism across trials, and recognize the saturation point where an eval suite stops distinguishing a good agent from a mediocre one.
Common Pitfalls and the Shape of a Working Eval System
A handful of pitfalls show up in almost every team: treating an eval as something you run once instead of a process you rerun on every change; over-optimizing against the eval set itself until the number looks great but the real product hasn't improved; reaching for generic metrics like "helpfulness" instead of building criteria around your product's actual failure modes; and trusting an LLM judge from day one without ever checking it against a human.
Strip away the specific techniques, and a working eval system does exactly three things well: it forces a look at real data before guessing, it picks the cheapest grader still trustworthy enough for each kind of question, and it reruns consistently every time something changes. Get those three right, and a team stops having to guess whether the latest update is actually better - they have a number that tells them.