How AI 'Understands' Requests
Attention Mechanism & Weighting, how word choice shifts tone, and boundaries between Parametric Knowledge and Ground Truth Verification Layers.
Model "Understanding" via Attention Weights and the Boundaries of Ground Truth
The biggest pitfall for PMs is assuming AI "reads and understands" like a human. In reality, AI "understanding" is pure mathematics: the Attention Mechanism calculating correlation weights between words.
Example: In EcoCart, understanding Attention clarifies why a small typo can derail entire outputs and why models cannot self-verify facts.
1. The Nature of AI "Understanding": Attention Weighting Mechanics
Humans read sequentially and connect text to real-world experience. LLMs process all tokens simultaneously: before generating each token, Attention calculates weights to identify which prompt words are most relevant.
"Understanding" is not human comprehension, but probability matrix computation matched against training patterns to select the next token.
Humans read in order; a model weighs everything at once
That difference is why one swapped word can pull an entire output off course.
Sequential, left to right
Ties each word's meaning to real-world knowledge and context
Fills in typos and unstated intent automatically
Parallel, all tokens at once
Assigns an Attention Weight to every token in the prompt
Processes exactly what you typed - nothing more, nothing less
2. How Input Phrasing Shifts Tone and Direction
Models do not infer unstated intent; they weigh the literal tokens you type. A small phrasing change shifts downstream output entirely.
Example: A user intends to type "formal" ("trang trọng") but typos it as "luxurious" ("sang trọng"). A human colleague easily infers a sincere apology. A model instead assigns high attention weight to "luxurious", shifting the apology into opulent, high-society language ("elite caliber", "high society") - completely unfit for late delivery support.
One word, redirected attention, a different email entirely
The model doesn't ask “did you mean formal?” - it weighs the literal token you typed and lets that weight steer everything downstream.
Line thickness = attention weight on that token
“Dear valued customer, we sincerely apologize for the inconvenience and appreciate your patience.”
3. Limits of Parametric Knowledge and Hallucination Risks
A model's memory is Parametric Knowledge - mathematical weights compressed from training. It cannot distinguish objective ground truth from popular linguistic patterns.
- Common Fact: "How tall is Mount Fuji?" → Correctly returns 3,776m due to high frequency across training data.
- Internal / Specific Fact: "What year was the founder of a new 10-person EcoCart startup born?" → Confidently invents a year (e.g. 1988) based on tech founder age patterns, lacking actual records.
Product Exercise: A seller enters "Write a description for new running shoes" with no technical specs:
- The model generates copy using generic industry patterns ("cushioned rubber sole", "breathable air mesh").
- PM Insight: These claims are hallucinations based on general patterns, completely untrustworthy as verified selling specifications.
4. The Boundary Between Language Prediction and Ground Truth Verification
To protect product safety, system architecture must decouple into 2 distinct layers:
| Component | Generative Layer (LLM Model) | Ground Truth Verification Layer (DB / RAG / APIs) |
|---|---|---|
| Objective | Fluid phrasing and brand tone synthesis | 100% accurate factual data retrieval |
| Nature | Probabilistic pattern inference | Deterministic structured data lookup |
| EcoCart Example | Generating: "Your order #12345 is being shipped by..." | Logistics API lookup: "ViettelPost - Tracking ID VP987" |
PM Rule: Never allow models to recall or guess factual data (pricing, balances, inventory, policy terms) - retrieve them from deterministic databases and inject them into Prompt Context (Context Grounding).
Design the Full EcoCart Refund AI Copilot - From Rule/Model Boundary to Ground Truth Layer
Module 1 Capstone: Design the complete EcoCart Refund AI Copilot for support agents:
- Rule/Model Boundary (Lesson 1): Separate deterministic business rules (return windows, order status) from generative model tasks (reply drafting).
- Automation Level (Lesson 2): Select Level 1 (Human-in-the-Loop), designing UI that clearly distinguishes AI suggestions from agent edits.
- System Prompt (Lesson 3): Author all 4 components (Task, Context with variables
{customer_name},{policy_text}, Constraints, Response Examples). - Ground Truth Layer (Lesson 4): Inventory all factual data points (order ID, balance, policy rules) required from Database/APIs.
- Adversarial Testing: Run 3 edge-case inputs (incomplete info, ambiguous request, adversarial prompt) to verify system resilience.