Choosing The Right Human-in-the-Loop Stopping Point By Risk
Understand HITL as a 4-point spectrum (review-before-execute, execute+undo, sample-based, no human) rather than an on/off flag, and choose a stopping point per branch based on Cost of Failure.
Choosing The Right Human-in-the-Loop Stopping Point By Risk
AI Literacy Lesson 1 already introduced "Hybrid": when Cost of Failure is high → Rule-based, or Hybrid with Human-in-the-Loop (HITL). That was the architecture decision. Lesson 31 also touched it under the "wrong but confident" failure type - "block upstream with an output-checking rule." This lesson separates it out to answer a concrete UI question: where in the flow does a human stand, and what are they allowed to do there?
Running example: HireFlow - an AI tool that screens candidate resumes, tagging each CV "fit / not fit / needs a closer look."
1. HITL isn't an on/off flag - it's a spectrum with multiple stopping points
The most common mistake: treating "has HITL" as a binary flag. In reality it's a spectrum with several stopping points, each trading speed for safety differently:
- (a) Review-before-execute - the AI proposes, and the action only actually happens once the user clicks "approve." Safest, slowest.
- (b) Execute-with-easy-undo - the AI executes immediately, but leaves a cheap, clear way back (a prominent undo button, a change log). Faster than (a), acceptable when both the cost of being wrong and the cost of fixing it are low.
- (c) Sample-based review - not reviewing every item, but periodically pulling a random sample (or a low-confidence sample) for batch human review, used to tune thresholds/the model over time.
- (d) No human in loop - only used when Cost of Failure is low and the AI has proven stable reliability over time (usually starts at (a)/(b) and works down to (d), never starts here).
Which stopping point to use isn't fixed per product - it's fixed per decision branch within the same product - and connects directly to the confidence threshold from Lesson 29: high confidence → can drop to (b)/(d); low confidence → forced to stay at (a).
HITL Is A Spectrum With 4 Stopping Points, Not An On/Off Flag
Each point trades speed for safety differently - choose per branch based on Cost of Failure.
Slowest
Highest
High Cost of Failure, low/medium volume.
2. Example: HireFlow splits into 2 branches by risk level, instead of one shared stopping point
Original design: the model auto-removes CVs tagged "not fit" from the pipeline, and the recruiter only sees what's left. Problem: this is a high-Cost-of-Failure decision (wrongly rejecting a strong candidate means losing a hire, and can carry discrimination-related legal risk), yet it was running at level (d) - no one reviewing it.
Fix: CVs tagged "not fit" are not removed from the pipeline - they move into a separate "AI suggests reject - needs confirmation" tab, and the recruiter has to actively review them (moving this specific high-risk branch from (d) to (a)). The "fit" branch still runs at level (b) - auto-advanced into the main pipeline, with a visible "remove from shortlist" button if the recruiter disagrees after reviewing.
3. The four stopping points and when to use each
| Stopping point | Speed | Safety | Fits when |
|---|---|---|---|
| (a) Review-before-execute | Slowest | Highest | High Cost of Failure, low/medium volume |
| (b) Execute + easy undo | Fast | Medium | Medium Cost of Failure, cheap and fast to fix mistakes |
| (c) Sample-based review | Fast (doesn't block each item) | Medium, long-term oversight | Volume too large to review item by item |
| (d) No human in loop | Fastest | Lowest | Low Cost of Failure, the model has proven stable |
Important caveat: (a) isn't always "safest so always better" - applying (a) to an entire product regardless of Cost of Failure exhausts the user with having to approve everything, leading to rubber-stamping. At that point "having a human in the loop" is just a formality, with no real safety effect left.
4. Analogy: an editor-in-chief allocates oversight by the risk of each piece
It's like a newspaper editor-in-chief working with interns at different levels. A sensitive piece (politics, legal) - the editor reads and approves line by line before it publishes (a). A routine piece - it publishes right away, but an editor can pull/fix it fast if readers flag something wrong (b). With hundreds of short translated news items every day - not reading all of them, just pulling a random sample each week to gauge overall quality (c). No editor-in-chief applies the same level of control to every kind of piece - they allocate attention by risk.
Exercise 32.1: RefundBot - AI that auto-processes refund requests for an e-commerce marketplace, split into 2 branches: (1) orders under 200k VND, with a reason that clearly matches policy (defective item, wrong item shipped); (2) orders over 2 million VND, or a vague reason ("not as expected"). Currently both branches run at level (a) - support staff must approve every single case. Propose which of the 4 stopping points each branch should use, explaining your reasoning based on Cost of Failure.