EvaluationLLM EvalsBenchmarkingAI Agent Skill
llm-evaluation
Create synthetic evaluation benchmarks, grading rubrics, and automated judge harnesses for AI models.
Install Options
Install into Claude Code, Cursor, Codex, or Antigravity with:
$ npx skills add wshobson/agents@llm-evaluation
Real Work Situation Solved
Build an AI eval benchmark from scratch with zero sample data
“Engineers ask "how do we measure model accuracy?" before launch - where do you get your first test cases?”
5-Step Action Framework
- 1Handcraft 20 core user scenarios (10 standard asks, 5 ambiguous/adversarial probes, 5 out-of-scope queries).
- 2Define the ground truth target answer and a pass/fail rubric for each test case.
- 3Have 2 teammates evaluate the test independently - if they disagree, your rubric isn't objective enough yet.
- 4Run the model through all 20 cases and record the baseline pass rate before tuning prompts or RAG.
- 5Append a new test case every time you find an unexpected failure or weird output during dogfooding.
Before
"Wait until launch and collect real user queries to evaluate."
After
"Start with 20 curated golden test cases and clear rubrics (10 happy + 5 edge + 5 adversarial), measure 65% baseline pass rate, target 85% for beta release."
Related Course Lesson
AI Agents for PM • Lesson 11
SKILL.md Source Instructions
SKILL.md • Read-only preview# LLM Evaluation Skill Construct golden datasets, establish multi-dimensional scoring rubrics (Relevance, Groundedness, Safety), and automate benchmark runs.