Module 5 • Lesson 46• 35 mins

Measuring Agentic Product Quality in Production

Monitor and diagnose agent operational health through the metric triad: Task Completion Rate, Average Steps to Completion, and Intervention Rate.

Master the definitions and diagnostic roles of the Metric Triad

Perform root-cause diagnosis by reading the three metrics holistically

Measuring Agentic Product Quality in Production

In AI Literacy Lesson 13, you learned the 3 pillars of AI evaluation: Eval Dataset, Rubric, and Pass Rate — built to evaluate the accuracy of isolated, single-turn outputs. However, when deploying multi-step Agentic systems into enterprise production, the atomic unit of evaluation shifts: it is no longer "a single chat response", but an end-to-end multi-step Workflow Run.

Running example: RefundBot — analyzing a monthly production telemetry report across 500 automated e-commerce refund transactions.

1. The Core Agentic Operations Metric Triad

To assess the operational health of an agentic product, PMs must monitor 3 interdependent metrics:

Task Completion Rate: Percentage of workflow runs satisfying the Overall Completion Criteria (Lesson 38) out of total invocations. Note: This is not "% of internal steps executed", but verified business goal resolution.
Average Steps to Completion: Average sequential steps taken by the agent to reach a successful conclusion (including Retry attempts from Lesson 43). An upward drift over time indicates Systemic Regression or third-party infrastructure degradation.
Intervention Rate: Percentage of runs pausing for human authorization (Pause) or triggering manual triage handoffs (Human Handoff, Lesson 44). This acts as the direct operational audit against your declared Autonomy Level (Lesson 42).

The Agentic Operations Metric Triad

Inspect RefundBot's 500-run production telemetry to diagnose systemic root causes.

Select an operational metric:

3. Intervention Rate

Critical Alert (Support overload)

PM Operational Definition: Percentage of runs entering Pause approval states or triggering permanent Human Handoffs.

Target Baseline: < 5%

Production Telemetry: 15% (3x above acceptable ceiling)

⚠️ Chẩn đoán: Setting 'Notify After' autonomy on high-value orders caused frequent deadlocks, forcing support staff to manually triage.

Root-Cause Principle: Always Read All 3 Metrics Together

An 82% Completion Rate looks passable in isolation, but an inflated 6.2 Avg Steps (excessive retries) and a 15% Intervention Rate (3x ceiling) proves Autonomy was set too optimistically, trapping the agent in deadlocks and burning support labor.

Never evaluate a single metric in isolation; cross-reference all 3 metrics simultaneously to evaluate true agent operational health.

2. Mandatory Rule: Always Read All 3 Metrics Holistically

A dangerous product anti-pattern is evaluating single metrics in isolation. Examine RefundBot's real-world 500-run monthly telemetry:

Metric	Production Actual	Target Baseline	Variance Evaluation
Task Completion Rate	82%	≥ 90%	Below standard (-8%)
Average Steps	6.2 steps	≤ 5.0 steps	Inflated bloat (+1.2 steps)
Intervention Rate	15%	< 5%	3x above allowed ceiling

Root-Cause Diagnosis:

In isolation, an 82% Task Completion Rate sounds acceptable ("we automated over 8 out of 10 requests").
Cross-referencing against 6.2 Average Steps reveals the agent is burning compute in retry loops at the payment gateway API step.
Combining this with a 15% Intervention Rate (3x ceiling) proves that Autonomy Levels were set too optimistically (granting Notify After where Human Approval was required), repeatedly backing the agent into deadlocks and dumping crisis tickets onto human customer support.

3. Metric Pathology & Remediation Matrix

Telemetry Pattern	Root Cause Diagnosis	PM Remediation Action
Low Completion Rate, Normal Steps	Permissive Step DoDs; logic errors breaking Overall Criteria	Tighten deterministic validation gates at core steps
High Completion Rate, Spiking Steps	Agent trapped in retry loops; flaky external dependencies	Enforce strict Retry Budgets, add circuit breakers
Spiking Intervention Rate	Autonomy Level set too aggressively; missing tool scopes	Lower Autonomy tier or expand Minimal Tool Surface
High Completion Rate, 0% Intervention	Risk of "Silent Failures" (swallowing errors without logging)	Audit error boundaries, guardrails, and audit logs

4. Analogy: Restaurant Kitchen Operations Report

Monitoring the Metric Triad is identical to an executive chef evaluating kitchen efficiency:

Task Completion: Measures the percentage of diners who received their ordered dish and left satisfied.
Average Steps: Measures how often line cooks had to refire a dish due to burning or bad seasoning (Retries).
Intervention Rate: Measures how often the floor manager had to visit tables to apologize and resolve food complaints (Human Handoff). A restaurant with 95% dish delivery where the manager must apologize to 20% of tables is operational chaos masked by vanity metrics.

Exercise 46.1: You manage OnboardingBot — an agent configuring tenant workspaces, setting user roles, and seeding sample data for new B2B SaaS customers.

Month 1 Telemetry:

Total workflow runs: 1,000 onboarding instances.
Task Completion Rate: 74% (Target ≥ 92%).
Average Steps: 8.4 steps (Standard happy path is 5 steps).
Intervention Rate: 22% (Target ≤ 8%).

Your Task:

Analyze OnboardingBot's operational health across the 3 metrics.
Formulate 2 root-cause hypotheses explaining this pathology.
Propose a concrete remediation plan to return metrics to target baselines.