AI's Limits: When to Doubt
Why hallucination hides behind confident tones, 4 signals that trigger verification, and independent source cross-checking workflows.
AI's Limits: Knowing When to Doubt Instead of Trusting Outright
In Module 1, we learned AI models can hallucinate because they fundamentally predict next tokens without internal ground truth verification. This lesson answers the key PM question: How do you spot when to stop and verify rather than blindly trusting AI outputs?
1. Why Hallucination Is Dangerous: Confidence Does Not Equal Correctness
Hallucination is insidious because it never reveals itself through tone. Models generate fabricated facts with the exact same fluent conviction as verified truths - unable to distinguish between "certainty" and "guessing."
Analogy: The model operates like an articulate, charismatic salesperson who speaks with total conviction - yet eloquence does not equal factual expertise.
2. Four Suspicious Signals That Trigger Verification
Because tone cannot be trusted, PMs must recognize 4 high-risk triggers:
- Specific, Uncommon Details: Exact dates, statistical figures, obscure names, source URLs.
- Recent Events: Any event occurring after the model's Knowledge Cutoff.
- High-Stakes Decisions: Financial calculations, legal compliance clauses, medical dosages.
- AI Lacks Data Access: Queries about private internal data or events never provided in the prompt.
Four signals worth stopping to verify
Select a signal to see where - if anywhere - it shows up in the journalist's example passage.
“This pottery village was founded in 1421 by master craftsman Trần Văn Tứ, who trained in China under the Ming dynasty. In 1978, the village won the national Golden Hands award.”
Select a signal above to highlight it in the passage.
3. Error Risk Concentrates on Hard Facts and Named Entities
In the pottery village example above, the writing is completely fluent, but hallucination risk concentrates entirely on hard entities: the founding year (1421), craftsman's name (Trần Văn Tứ), and award title (Golden Hands 1978).
PM Rule: Polished prose is the model's strength; factual accuracy of numbers and entities is its blind spot.
4. Verification Workflow: Cross-Check Independent Sources (Never Re-prompt)
When encountering suspicious signals, never ask the AI "Are you sure?" - it simply regenerates affirmative answers with the same misleading confidence.
Asking again doesn't help - checking an independent source does
“Are you sure?”
Same confident tone, same answer - nothing about how it generates a reply has changed.
Cross-check an independent source
The original document, a search, or someone with real expertise - something the model's own confidence can't substitute for.
Standard Protocol: Validate directly against internal databases, original source records, or human subject-matter experts before publishing.
Design the EcoCart AI Support & Policy Assistant - From Token Costs to Suspicious Signals
Module 2 Capstone: Design a bilingual (Vietnamese/English) Policy Assistant for EcoCart:
- Token Optimization (Lesson 5): Choose System Prompt language (Vietnamese vs. English), evaluating the trade-off between ~35% token savings and translation nuance risks.
- Context Window (Lesson 6): Identify immutable policy invariants (return windows, warranty terms) pinned into the System Prompt to prevent context drift in long sessions.
- Suspicious Signals (Lesson 7): Create a 4-signal verification checklist for support agents before delivering AI responses to users.
- Adversarial Testing: Run 3 edge cases (token-heavy query, 50+ turn conversation, high-risk hallucination trap) to validate architectural safety.