Managing Latency And Perceived Performance Through Streaming
Compare 3 latency-masking techniques - static loading, skeleton state, streaming - and pick the right one by output shape, not by default because it's an AI feature.
Managing Latency And Perceived Performance Through Streaming
Lesson 27 flagged it: the second axis that breaks once a model sits behind a feature is response time. Worth being precise here: rule-based systems don't have "perfectly fixed" latency either - it still drifts with system load, network conditions, data size - but that drift sits inside a narrow, predictable band (say, 200-300ms), tight enough that a static loading indicator stays accurate almost every time. A model - especially a text-generating LLM - has no such narrow band: a simple question answers in 0.5s, a complex one can take 8-10s. The problem isn't "slower" - it's the huge swing between calls, which makes any fixed loading pattern wrong for at least half the cases.
Running example: LegalDraft - an AI tool that drafts contract clauses on a lawyer's request.
1. Three techniques, from weakest to strongest
- Static loading indicator - a spinner/progress bar that doesn't change with actual processing. Still fine for fast, stable tasks (under 1-2s, a narrow band like rule-based systems have), but with an AI feature that has large latency variance, a spinner frozen for 8 seconds makes the app look like it crashed.
- Skeleton state - show the "skeleton" of the result before real data arrives. Reduces the feeling of waiting because the user sees structure before content - but it's still waiting, and doesn't solve a 30-second AI response.
- Streaming response - display output as the model generates each piece (the same mechanism as AI Literacy Lesson 2's Streaming UI). It turns "waiting time" into "reading time," and more importantly: the user can interrupt/cancel mid-generation if the answer is heading in the wrong direction.
3 Latency-Masking Techniques, Weakest To Strongest
LegalDraft swapped a static spinner for streaming - same processing time, completely different feeling of waiting.
Turns waiting into reading; lets the user interrupt early.
Needs backend/API support for chunked output; not every model has this.
2. Example: LegalDraft swaps a static spinner for streaming, duplicate requests drop sharply
Original version: click "Generate clause" → static spinner → after 6-12 seconds (depending on complexity) a full block of long text appears. Lawyer feedback: the app felt "frozen," and many people clicked the button again thinking the request hadn't gone through - creating duplicate requests and wasted API cost.
Switched to streaming: text appears sentence by sentence as the model generates it. Same actual 6-12 second processing time, but the feeling of waiting disappeared because the lawyer started reading from second 1. A good side effect: 2 lawyers reported hitting stop mid-generation when they saw the model going off-topic, saving both reading time and cost - something impossible with a non-streamed response.
3. Choose the technique by output shape, not by default because it's an AI feature
| Technique | Solves | Doesn't solve |
|---|---|---|
| Static spinner | Fast, stable tasks (under 1-2s) | Large latency variance - looks like the app froze |
| Skeleton state | Reduces the "empty" feeling while waiting | Doesn't shorten the feeling of waiting for genuinely long tasks |
| Streaming | Turns waiting into reading; lets the user interrupt early if output is off-track | Needs backend/API support for chunked output; not every model has this |
Note: streaming isn't free. With tightly structured data (JSON, numeric tables), streaming piece by piece can make the UI "jump" repeatedly or show incomplete data that misleads. Whether to stream depends on output shape, not a default rule that every AI feature must stream.
4. Analogy: a closed kitchen vs. an open kitchen
A static spinner is like standing in front of a restaurant's closed kitchen door, with no idea how far along the dish is - 10 minutes and 2 minutes feel identical, just growing impatience. Streaming is like a restaurant with an open kitchen: you see the chef cutting, stir-frying, the dish being plated bit by bit - the same 10-minute wait, but it feels completely different because you can see progress.
Exercise 30.1: FinCheck - an AI app that analyzes company financial reports, returning long-form commentary plus an extracted numeric table (not just text). Processing time varies from 3-15 seconds depending on report length. Pick 1 of the 3 techniques above (or a combination) for the text commentary and for the numeric table, and explain why these two parts might need different treatment.