Designing Error States And Graceful Degradation
Classify the 3 AI-feature failure types (no-result, wrong-but-confident, partial) and design a distinct recovery path for each.
Designing Error States And Graceful Degradation
Lesson 29 set a threshold: below the low-confidence line → "don't auto-execute, ask again or route to a reviewer instead." This lesson goes deep into that exact branch - when the model is wrong or can't find an answer, what does the UI do? Not just "show a signal," but design the whole recovery flow.
Running example: InvoiceBot - AI that automatically extracts data from scanned invoices (vendor, amount, due date) and auto-fills the accounting system.
1. Three failure types that need separate designs, because recovery differs
Rule-based software has error states that are discrete and fully enumerable (404, validation failure, timeout). AI features have a kind of error that fits none of those categories: the model responds "successfully" from a technical standpoint (no crash, no timeout) but the content is wrong or off-topic - there's no exception to catch, only detection through low confidence (Lesson 29) or user feedback afterward.
- No-result - the model doesn't have enough grounds to answer. The easiest type to handle, because the system knows it doesn't know. The UI should say "couldn't find this information" outright rather than force an answer - forcing an answer is the most harmful behavior, because the user can't tell it apart from a grounded one.
- Wrong answer that looks confident - the most dangerous type, since the model doesn't know it's wrong. Can't be solved by pure error-state design - it needs to be blocked upstream with a confidence threshold (Lesson 29) or an output-checking rule (Hybrid - AI Literacy Lesson 1).
- Partial/inconsistent - right on the main point but wrong on a small detail. The hardest to design for since there's no clean right/wrong boundary - the common approach is to highlight just the suspect part (based on per-segment confidence) rather than flagging the whole answer.
3 AI-Feature Failure Types, Each Needing A Different Recovery
No exception to catch - only detected through low confidence or user feedback afterward.
No
Block upstream with a confidence threshold + an output-checking rule.
An extracted amount doesn't match the expected format (negative, abnormally large) → block auto-save until the user confirms.
2. Example: InvoiceBot moves error detection from month-end back to entry time
Original version: the model extracts something wrong → the bad data goes straight into the books, and the accounting team only catches it at month-end reconciliation, having to dig back through hundreds of invoices to find which one was mis-entered.
Redesigned around the 3 types: (a) an invoice too blurry to read the numbers → don't auto-fill, flag "needs manual entry" right away (no-result); (b) an extracted amount doesn't match the expected currency format (negative, abnormally large) → block auto-save until the user confirms (wrong-but-confident); (c) the vendor name is correct but the address is partly wrong → save it, but highlight just the address field, don't block the whole invoice (partial). Errors still happen at the same rate, but the detection point shifted from "month-end" to "right at entry" - where the cost of fixing it is nearly zero.
3. The one principle that runs through all 3 types
| Failure type | Does the system know on its own | Recovery direction |
|---|---|---|
| No-result | Yes (knows it lacks grounds) | Say "not found" outright, don't force an answer |
| Wrong but looks confident | No | Block upstream with a confidence threshold + an output-checking rule |
| Partial/right in part | Partially (per-segment confidence) | Highlight just the suspect segment, don't flag the whole answer |
Whatever the type, one principle runs through all of them: undo/edit must be cheaper than discovering the error after it's already been acted on. Since AI errors can't be predicted ahead of time, the cost of fixing a mistake after the user already trusted it and acted on it is always far higher than designing a "review before commit" step from the start.
4. Analogy: the honest new hire vs. the confidently wrong one
It's like the difference between a new hire who says plainly "I'm not sure about this part, let me check" (type a - harmless, just a beat slower) and one who says something wrong with total confidence (type b - the most dangerous, since no one thinks to double-check). A good manager doesn't try to "fix the personality" of employees - they design a review process that catches type-two mistakes before they reach the customer.
Exercise 31.1: ShiftPlanner - AI that auto-schedules store staff shifts based on time-off requests plus foot-traffic forecasts. Currently: when the model lacks enough forecast data for a given day, it still fills in a schedule based on an "average guess" without flagging anything - the store manager only finds out when that day is severely understaffed. Identify which of the 3 failure types this is, and propose 1 concrete UI change so it no longer "looks confident" while actually guessing.