Error Analysis: Read Real Traces Before You Build an Evaluator
The most common eval mistake isn't a missing tool - it's writing an evaluator before knowing where the product actually breaks. Error analysis means reading real traces, labeling them by hand, and letting the data show which failures to measure.
Part of: LLM Evals: Measuring AI Quality Before It Quietly Breaks Your Product
A team ships an LLM feature, writes an evaluator on day one to "cover the basics," and watches it stay green for weeks while user complaints keep trickling in. The evaluator isn't broken - it's just grading the wrong thing, because nobody read what real users actually typed and what the model actually produced before deciding what to measure.
Error analysis fixes the order of operations: read real traces first, let the failures you actually see define what an evaluator should check, and only then write the assertion or rubric.
From Raw Traces to a Targeted Evaluator
Why Writing an Evaluator Before Reading Traces Always Backfires
Most eval suites start from intuition: a PM or engineer lists the failure modes they assume matter - "make sure it's polite," "make sure the JSON is valid" - and writes checks for those. The checks pass, the dashboard looks healthy, and the team ships with confidence. Then real usage surfaces a failure nobody wrote down: a tone that reads as dismissive under a specific kind of complaint, a booking agent that reports success while the underlying reservation is wrong. An evaluator can only check for failure modes someone already knows to look for - it can't discover the ones nobody anticipated. Reading real traces before writing a single assertion is the only way to trade assumed failure modes for actual ones.
The Process: From Raw Transcript to a Pass or Fail Call
The mechanics are unglamorous on purpose. Pull a batch of real transcripts - a mix of recent traffic, not hand-picked "interesting" ones, since cherry-picking just re-introduces the same assumptions you're trying to escape. Read each one end to end: the input, the full output, any tool calls, the final state left behind. For each trace, ask one question - would the user consider this a good outcome? - and write down a binary verdict plus a one-line reason. The reason matters more than the verdict: "fail" alone tells you nothing to act on, while "fail - cited the policy correctly but never acknowledged the damaged item" points straight at what needs to change.
Binary Labels, Not a Rating Scale
A 1-5 scale feels more precise than pass/fail, but in practice it mostly produces arguments over where a 3 ends and a 4 begins - debate that consumes review time without producing an actionable signal. A binary label forces a decision: is this good enough to ship as-is, or not? That decision is exactly what a shipping team needs, and it's far easier to keep consistent across a reviewer's 30th trace and their 3rd than a five-point scale ever is.
Who Owns the Definition of "Good"
Every feature needs one person - not a rotating committee - who owns what "good" means for it. Splitting that ownership across multiple reviewers without a shared, explicit standard produces labels that drift: the same trace gets called a pass by one reviewer and a fail by another, and the resulting taxonomy is noise instead of signal. That owner doesn't have to personally read every trace forever, but they set the standard the rest of the team calibrates against, and they stay close enough to the raw data that the standard doesn't quietly rot.
Reading Traces Builds a Taxonomy, One Label at a Time
Each trace gets a binary call and a one-line reason. Read enough of them, and the same few failure shapes start repeating — that repetition is the taxonomy.
Can I get a refund? The item arrived damaged.
Per policy 4.2, damaged-item claims require photo evidence submitted within 48 hours of delivery. Your request falls outside this window.
Technically correct policy citation, but reads as cold — no acknowledgment the customer got a damaged item.
Once a failure shape repeats, it graduates from a one-off note to a named category worth writing an evaluator for.
From Individual Labels to a Shared Failure Taxonomy
A single failing trace is a data point. The same shape of failure showing up across six, eight, ten traces is a category worth naming and worth writing an evaluator for. The naming matters: a category called "bad response" points at nothing, while "transcript reads correctly but the recorded outcome is wrong" tells the next engineer exactly what class of bug to build a check for. Building this taxonomy is the actual output of error analysis - the evaluators that come afterward are just automating the categories this reading process already found.
Open Coding vs. Closed Coding: When to Use Which
Qualitative research has a useful distinction here. Early in a reading pass - the first 15-20 traces or so - use open coding: no predefined categories, just freeform notes on what went wrong in each trace's own terms. Forcing traces into an existing bucket too early hides failure modes that don't fit the buckets you already have. Once a handful of categories start repeating, switch to closed coding: check each new trace against the existing list first, and only add a new category when something genuinely doesn't fit anywhere. That switch is what turns a pile of one-off notes into a taxonomy stable enough to build evaluators against.
Cost of Reading Traces vs. Cost of Skipping Error Analysis
Model scenario: reading and labeling the first 30 traces at $75/hour of reading time, surfacing most failure categories before an evaluator gets written.
~4/7 failure categories seen (55%)
+2 new categories (cumulative 85%)
No new categories — the signal to stop
How Many Traces Is Enough, and When to Stop Reading
The honest answer is: until new traces stop teaching you anything new. In practice, for a single well-scoped feature, the first 20-30 traces usually surface most of the recurring failure categories - after that, additional traces mostly confirm categories already found rather than reveal new ones. That saturation point, not a fixed number picked in advance, is the real signal to stop reading and start writing evaluators. Stopping too early means an evaluator that misses common failures; reading well past saturation mostly burns review time without changing what gets built.
Keep the Loop Open, Error Analysis Isn't a One-Time Pass
The traces that mattered at launch won't be the traces that matter after the next model upgrade, prompt rewrite, or a new segment of users showing up. Each of those changes shifts what "real" traffic looks like, which means the taxonomy built from the last reading pass can go stale exactly when it matters most. Treat error analysis as a habit that re-runs at every meaningful change, not a step that happens once during onboarding and then gets left behind - the loop between seeing a real failure and updating what gets measured is what keeps an eval suite honest.
Common Pitfalls in Error Analysis
A few mistakes show up again and again: jumping straight to an LLM-as-judge before any human has read a single trace, so the judge gets calibrated against nobody's actual standard; treating error analysis as a launch-week ritual instead of a recurring habit; handing labeling off entirely to a team disconnected from product decisions, which breaks the feedback loop that makes the reading worthwhile in the first place; and naming categories by symptom instead of mechanism, which produces a taxonomy nobody can act on. Avoid those, and error analysis does exactly what it's supposed to: it replaces a guess about what's broken with a list of what's actually broken, in the words of the people who'll have to fix it.