Designing Explainability To Build Trust Without Cluttering The UI
Distinguish 3 product-layer explainability forms (citation, reasoning trace, influencing factors) and scale their use with Cost of Failure instead of switching them on by default.
Designing Explainability To Build Trust Without Cluttering The UI
AI Literacy Lesson 1 covered it: a model is a "black box" - its output is the combined result of billions of parameters, hard to trace like an if-else chain. That's a technical limit, unchangeable. But this lesson doesn't ask "can the model explain itself" - it asks a different question, entirely at the product layer: does the UI need to build a separate explanation layer, apart from the model's real internal mechanism, so the user can still trust the result?
Running example: PolicyBot - a chatbot that answers employee questions about internal company policy (leave, benefits, procedures) based on an HR document store.
1. Three forms of explainability, differing in what they lean on
Explainability in UI is not exposing the model's real internal mechanism to the user - that's not feasible, and no one needs to know which vector fired. It's building a "product-layer explanation" that lets the user judge whether a result is trustworthy, without needing to understand how the model actually works.
- (a) Citation/source pointer - shows which specific data/document an output is based on. Strongest when available (especially in RAG systems - Module 3), because the user can verify independently of whether they trust the model.
- (b) Reasoning trace - replays (a simplified version of) the steps the model took to reach its conclusion, e.g. "Checked: order within the 7-day window ✓, status 'Delivered' ✓ → Eligible for refund." Useful when a conclusion depends on multiple conditions.
- (c) Confidence + influencing factors - doesn't explain how the conclusion was reached, just states what makes it more/less trustworthy, e.g. "This suggestion is based on 12 similar orders you've bought before." Lightest.
3 Product-Layer Explainability Forms
Not exposing the model's real mechanism - building a separate explanation layer so users can judge trustworthiness.
"Based on what?"
Needs source traceability - readily available in RAG systems, harder in pure text-generation models.
2. Example: PolicyBot adds citations - accuracy doesn't change, but trust does
Original version: a short text answer, no citations. The most common employee feedback: "I don't know if I can trust this, what if it's making things up" - even though the model was right most of the time, simply because there was nothing to verify against, trust stayed uniformly low across every answer, including correct ones.
Adding citations (form a): every answer links to the exact passage in the source HR document it's based on. The result wasn't that the model suddenly got more accurate (accuracy didn't change) - it's that employees could self-verify in 5 seconds instead of having to ask HR directly, and for the rare wrong answers, they caught them immediately because the cited passage didn't match the answer.
3. Three forms and the infrastructure each needs
| Form | Answers what question | Infrastructure needed |
|---|---|---|
| (a) Citation | "Based on what?" | Needs source traceability - readily available in RAG systems, harder in pure text-generation models |
| (b) Reasoning trace | "What was checked?" | Needs the logic rebuilt step by step - fits when a conclusion depends on multiple conditions |
| (c) Influencing factors | "Why is this trustworthy/not?" | Lightest - doesn't need reasoning steps separated out |
One easy-to-miss caveat: explainability isn't free in UI terms - too many citations, an overly long reasoning trace, turns a short answer into a cluttered block of text. For a low-Cost-of-Failure task, forcing the user to read "why this suggestion" just slows the experience down unnecessarily. Just like confidence signaling in Lesson 29, explainability should scale with the Cost of Failure of the decision, not be switched on by default for every output.
4. Analogy: "trust me on this" vs. pointing to clause 4.2
It's like the difference between an advisor who says "trust me, that's correct" and one who says "I concluded that because clause 4.2 of the contract states..." - both might be right with equal probability, but only the second gives you a way to check for yourself instead of relying entirely on whether you trust them.
Exercise 34.1: BudgetAdvisor - an AI app that analyzes personal spending, producing statements like "this month you spent 40% more on eating out than your 3-month average." Right now it only shows that exact sentence, nothing else. Propose which of the 3 explainability forms fits this use case - explain why that form suits this type of "compare figures over time" statement better than the other two.