Skill Triggers & Repeated Reliability pass^k
Skill triggers as the first gate, trigger confusion matrix, Exact/In_order/Any_order evaluations, and exponential reliability decay pass^k.
Evaluate the failure surface, not just the answer
A polished answer can hide a broken system. The wrong skill may have triggered, the right skill may have called an unsafe tool, or a newly installed skill may have degraded unrelated requests. The failure map below separates four surfaces that need different evidence.
The Agent Skill failure surface
A polished answer can hide a broken system. Four surfaces need four different kinds of evidence.
Trigger failure
Does the intended skill fire, and do adjacent skills stay quiet?
Positive and negative cases, confusion matrix, trigger accuracy target.
Trigger failure means the skill runs for the wrong request or fails to run for the right one. Execution failure means routing was correct but the procedure or tool trajectory was wrong. Token budget failure appears when the skill consumes enough context to degrade the rest of the session. Regression appears when one skill overlaps with another or changes routing across the library.
Treat those four surfaces as the minimum definition of complete eval coverage:
Use the refund skill to see why the surfaces must be tested separately. First, send positive and negative payment requests and verify the routing decision: that is trigger evidence. Second, for each correct activation, verify the refund explanation and the read-only tool sequence: that is execution evidence. Third, add the skill to the existing payments library and rerun delivery, declined-card, and refund-status cases: that is regression evidence. Fourth, repeat those unrelated cases with the realistic conversation history, tool schemas, and 5–15 commonly active skills: that is token-budget evidence. A failure in any step identifies a different owner and fix.
The table is the release recap after those four experiments:
| Surface | Release question | Metric or artifact |
|---|---|---|
| Trigger | Does the intended skill fire, and do adjacent skills stay quiet? | Positive and negative cases, confusion matrix, trigger accuracy target |
| Execution | Is the result correct, and was it reached safely? | Output rubric, expected tools, trajectory assertions, tool unit tests |
| Regression | Did the new skill break an existing capability? | Full-library baseline comparison with zero accepted drops |
| Token budget | Does realistic co-loading degrade unrelated work? | Token footprint, latency/cost delta, unrelated-task quality under 5–15 co-loaded skills |
A passing happy path does not compensate for a missing surface. If token-load or regression evidence is absent, the status is not evaluated, not “probably ready.” The PM artifact should be a small coverage matrix with an owner, test set version, threshold, result, and evidence link for every row.
Those four surfaces say what must be covered. The evaluation toolkit says when and how evidence is accumulated. Begin with eval-as-unit-tests on every edit. Add a once the expected business behavior is stable. Use a calibrated LLM-as-Judge to scale output review, but retain human-rated cases. Before granting write authority, attack the boundaries and permissions with . Finally, observe a or small against real traffic before full release. No single tool replaces the previous stage.
The five patterns recap as follows:
| Pattern | What it proves | Required evidence / tier |
|---|---|---|
| Eval-as-Unit-Test | Expected behavior stays stable on every change | CI cases for routing, tools, output, and regressions; every skill |
| Golden Dataset | Representative inputs retain agreed outputs and trajectories | Versioned input/expected pairs; draft tier and above |
LLM-as-Judge | Rubric-scored output quality scales beyond manual review | Human-calibrated rubric and position swap; read-only and draft |
| Adversarial / Red-Team | Rephrasing, boundary, injection, permission, and recovery failures are exposed | At least one rephrase and negative boundary for every positive; before action-allowed |
| Canary / Shadow Mode | Offline behavior survives controlled production conditions | Shadow comparison or low-percent canary with alerts and rollback; before each action-allowed release |
The release packet should link all five artifacts and state which are not applicable. “We ran an eval” is too vague to audit.
The evaluation escalation ladder
Each stage accumulates evidence the previous one cannot produce. No single tool replaces the stage before it.
Eval-as-Unit-Test
Expected behavior stays stable on every change.
CI cases for routing, tools, output, and regressions; required for every skill.
Benchmarks explain why this rigor matters. SkillsBench found 19% of 84 real-world agent tasks performed worse with a skill than without one. Another production analysis found 56% non-invocation for skills expected to fire; stripped-down skill instructions scored 58% versus 63% for the base agent, while a passive AGENTS.md convention index reached 100% versus a 53% baseline. These results do not mean skills fail generally. They show that a narrow procedure, global convention, and retrieval layer are different product primitives, and a poorly placed skill can subtract capability.
The first two can often be observed in a single request. The last two emerge only when the real library and realistic context are loaded together. A demo that exercises one skill in isolation cannot establish production readiness.
For PMs, this changes the acceptance criteria. "The answer looks good" is one signal, not the release gate. The team must know which skill fired, which resources loaded, which tools ran, what state changed, and whether adjacent skills still behave correctly.