Rapid-Prototyping The Experience Before Building The Real Backend

Rapid-Prototyping The Experience Before Building The Real Backend

The last nine lessons were all design decisions: confidence signaling, steerability, error states, onboarding... But one practical problem gets in the way of testing those decisions: building a real AI feature (data pipeline, model integration, backend logic) takes far longer than building a plain CRUD form. If a PM waits until the backend is fully built to test UX, the mistakes covered in Lessons 27-35 (static spinners, rigid accept/reject, no fallback...) only surface after all that build effort - too late to fix cheaply.

Running example: back to MedSort (Lesson 29) - the emergency-priority suggestion system at a clinic.

1. Three techniques, from most fake to least fake

Rapid prototyping for an AI feature separates the experience from the real engine behind it, so you can test how the product feels before that engine even exists:

• (a) Wizard of Oz - the UI looks like a real AI feature, but behind it a real person is typing/selecting answers in real time (usually through a hidden control panel). Test users don't know they're talking to a human. Strongest for testing interaction (conversation flow, how clarifying questions are asked, perceived response speed) with zero lines of model integration.
• (b) Mock response - build the real UI, but return pre-written responses (fixed, or randomly picked from a written sample set) to test UI patterns on their own: what streaming looks like, how a confidence badge displays, whether steerability buttons feel smooth - completely separate from "does the model actually answer correctly."
• (c) Thin LLM wrapper - call an off-the-shelf model directly through the API with a hand-written prompt, skipping the complex data pipeline/retrieval a production version would need. More realistic than (a) and (b) since the output is genuinely model-generated, but not yet the final system since it lacks real retrieval/data.

2. Example: MedSort runs 3 stages before investing in a real data pipeline

Before investing in a real data pipeline (collecting and labeling thousands of case records), the team ran a 3-stage prototype: (1) Wizard of Oz for 1 week - a doctor sat behind a screen reading symptom descriptions nurses typed in, typing out priority level + confidence themselves, to figure out which interaction flow nurses found useful; (2) Mock response - built the 3-tier confidence UI (designed in Lesson 29) with fixed fake data, to test specifically whether nurses read/understood the meaning of the 3 color tiers correctly, separate from whether the model itself was accurate; (3) Thin LLM wrapper - called a general-purpose language model directly with a prompt describing the problem, without any specialized medical data yet, to see how good the "raw" suggestions were. Only after confirming both the UX and the output quality were promising enough did the team start investing in a real data pipeline.

3. The three techniques aren't mutually exclusive - pick based on the question you're answering

The goal of all 3 techniques is to reduce the cost of learning something is wrong, early - not to produce a polished demo for leadership. A Wizard of Oz prototype that's enough to surface 2-3 serious UX flaws in one test session is worth far more than 2 weeks of building the real backend only to find that same flaw afterward.

4. Analogy: three rehearsal stages before staging a full play

It's like the 3 stages of trying out a play before a full production: first the director reads the lines aloud themselves to check whether the pacing works (Wizard of Oz) - no real actors needed yet. Then a real stage is built with a few fixed sample dialogue lines to test lighting and sound (mock response) - no finished script needed yet. Finally, real actors improvise their parts (LLM wrapper) - only now do you see the real, unpredictable "aliveness," but it's still not opening night.

Exercise 36.1: Back to RefundBot (Lesson 32). Before building the real system, pick 1 of the 3 prototyping techniques to test specifically the "orders over 2 million VND or a vague reason" branch - the branch already determined to need human-in-the-loop level (a) review-before-execute. Explain why the technique you chose fits the specific question this branch needs answered (not the under-200k branch).