Launch Decision Framework & User-Visible Rollback Scenarios
Anchoring launch gates to owners and quantitative thresholds, rehearsing user-visible rollback drills, and authoring evidence-based launch memos.
Define ownership, launch criteria, and rollback before pilot
Assign named owners before the launch recommendation is written:
Give every release decision a person, threshold, and response time
For the sales-agent pilot, product owns the supported internal cohort and draft-only boundary; the agent owner owns prompt and fallback versions; the MCP owner owns read availability and emergency disable; security owns email and PII policy; evaluation owns the frozen cases and judge calibration; operations owns alerts, cost, and the rollback drill. Naming teams without an on-call person, decision right, or response time leaves the failure unowned.
| Area | Owner must decide |
|---|---|
| Product and spec | Supported job, autonomy boundary, acceptance criteria, and cohort |
| Agent and prompts | Agent version, instruction changes, fallback, and quality regression |
| Tools and MCP | Contract, availability, permissions, data access, and emergency disable |
| Security and policy | Risk tiers, policy rules, incidents, exception expiry, and audit evidence |
| Evaluation | Dataset, rubric, judge calibration, thresholds, and failure taxonomy |
| Operations | On-call, observability, cost budget, rollout control, and rollback exercise |
Use a launch scorecard with hard gates, not an average that can hide a critical failure. Example pilot criteria are: zero successful prohibited actions across the adversarial set; 100% receipt coverage for approval-required actions; all critical BDD scenarios pass; intent and trajectory scores meet the agreed baseline margin; rollback succeeds in a rehearsal; p95 latency and per-task cost stay within budget; and every alert, tool, policy, and dataset has an owner.
Predefine rollback triggers such as one confirmed unauthorized external action, critical-data exposure, a sustained drop below the quality threshold, repeated circuit-breaker trips, or cost above the guardrail. The rollback playbook should disable write tools, return the product to read or draft-only mode, revoke credentials, preserve traces, notify owners, and verify user-visible state. Rehearse it before launch and record recovery time; an untested rollback plan is only a document.
Rehearse rollback as a user-visible product scenario
Run rollback as a product scenario before the pilot. Inject a policy-version defect that would allow direct CRM write, confirm the safety monitor or adversarial eval detects it, activate the kill switch, revoke the write credential, keep read-only briefing available, preserve affected traces, and show users that actions are temporarily draft-only. The exercise passes only when the system reaches the safe state within the target time and the team can identify which spec, policy, agent, and cohort versions were exposed.
What should ship next?
Six release scorecards. Choose the recommendation that the evidence can actually support.
Fixed launch gate
Hard gates - zero successful prohibited actions - complete receipts for approval-required actions - all critical BDD scenarios pass - intent and trajectory scores meet the baseline margin - rollback succeeds in rehearsal - latency and cost remain inside budget - every alert, tool, policy, and dataset has an owner No average can compensate for a critical safety failure.
Release proposal
Launch the sales-research agent to 25 internal account executives with approved read-only data and draft-only CRM proposals.
Scorecard evidence
All critical BDD cases pass, the adversarial set records zero prohibited actions, receipt coverage is 100%, rollback completes in four minutes, and latency and cost remain inside budget.
What is the defensible recommendation?
A strong average never cancels a critical failure. Scope the recommendation to the evidence, the safe state, and the tested cohort.