Agent EvaluationTestingGitHubAI Agent Skill
agentic-eval
GitHub's standard agentic evaluation framework for scoring agent reasoning steps, tool use, and task completion reliability.
Install Options
Install into Claude Code, Cursor, Codex, or Antigravity with:
$ npx skills add github/awesome-copilot@agentic-eval
Real Work Situation Solved
Build an AI eval benchmark from scratch with zero sample data
“Engineers ask "how do we measure model accuracy?" before launch - where do you get your first test cases?”
5-Step Action Framework
- 1Handcraft 20 core user scenarios (10 standard asks, 5 ambiguous/adversarial probes, 5 out-of-scope queries).
- 2Define the ground truth target answer and a pass/fail rubric for each test case.
- 3Have 2 teammates evaluate the test independently - if they disagree, your rubric isn't objective enough yet.
- 4Run the model through all 20 cases and record the baseline pass rate before tuning prompts or RAG.
- 5Append a new test case every time you find an unexpected failure or weird output during dogfooding.
Before
"Wait until launch and collect real user queries to evaluate."
After
"Start with 20 curated golden test cases and clear rubrics (10 happy + 5 edge + 5 adversarial), measure 65% baseline pass rate, target 85% for beta release."
Related Course Lesson
AI Agents for PM • Lesson 11
SKILL.md Source Instructions
SKILL.md • Read-only preview# Agentic Eval Skill Define task-completion rubrics for agent workflows, run repeatable evaluation passes, and track pass/fail trends across agent versions.