"Human in the Loop" Has Become Approval Theater
Every AI governance deck has the same slide. There's a box labeled AI, an arrow, and a box labeled Human Review. The room relaxes. Someone will be checking.
But almost nobody asks the follow-up question: checking with what information, in how much time, under what incentive to disagree? A checkpoint on a diagram is not oversight. It's a place where oversight is supposed to happen, and the difference between those two things is where a lot of risk quietly lives.
A checkpoint measures process, not judgment
The appeal of human review is that it's easy to count. You can point to the number of approval gates in a workflow and put it in a compliance report. What you cannot easily count is whether the reviewer was in any position to catch a bad output.
That's the gap. Teams optimize the thing they can measure — checkpoints — and assume the thing they actually want — informed refusal — comes bundled with it. It doesn't. A reviewer facing three hundred AI-generated recommendations a day, each of which has been right the last forty times, is not performing an independent judgment on number 301. They're clearing a queue.
This isn't a character flaw. It's a well-documented cognitive pattern with a name.
The evidence is not encouraging
Automation bias — the tendency to over-rely on automated output — has been studied for decades, and the findings are consistent enough to be uncomfortable.
A systematic review in JAMIA looked across studies of automated clinical decision support and found that in 5.2% of prescribing cases, clinicians switched a correct answer to an incorrect one after receiving automated advice. These are trained professionals with domain expertise, actively engaged in the task, and the machine's wrong answer still pulled a meaningful share of them off a right one. Lower experience predicted more switching. So did trust in the system.
The policy-level picture is worse. Ben Green surveyed 41 policies mandating human oversight of government algorithms and concluded they suffer from two linked flaws: people generally cannot perform the oversight function being asked of them, and because the requirement exists on paper, it legitimizes deploying flawed systems anyway. His phrase for what these policies deliver is "a false sense of security."
Notably, regulators know this. The EU AI Act's human oversight provision, which came into force this month for high-risk systems, explicitly requires that overseers be enabled "to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)." The law names the failure mode inside the safeguard. That's an unusual admission, and worth taking seriously.
A rubber stamp is worse than no stamp
Here's the part that makes this an argument rather than a caveat.
If a system runs with no human review, everyone knows the risk is unmitigated. It gets treated accordingly — narrower deployment, tighter scope, more monitoring downstream.
Add a review step that nobody can meaningfully perform and two things happen at once. The risk stays exactly where it was, and the organization's perception of that risk drops sharply. Worse, accountability relocates. The human who clicked approve now owns the outcome, despite never having had the information or time to evaluate it. You've manufactured a liability sink and called it a control.
Empty oversight isn't neutral. It's negative, because it buys false confidence with someone else's exposure.
The test: how often does the human say no?
The useful diagnostic isn't how many review gates you have. It's your override rate.
Pull the last hundred AI outputs that went through human approval. How many were rejected, edited, or sent back? If the answer is zero — or a rounding error — you do not have a control. You have a logging mechanism that records a person's name.
A real review step needs four things, and most deployed ones have at most two: the reviewer sees enough context to form an independent view, has enough time to actually form it, faces no penalty for slowing things down, and possesses the authority to stop the process outright. Strip any one of those and you're back to theater.
The honest move, when you can't provide all four, is to say so — and then narrow what the system is allowed to do until it's safe unsupervised. That's a harder conversation than adding an approval box. It's also the only one that changes the outcome.
Sources: Automation bias: a systematic review of frequency, effect mediators, and mitigators (Goddard, Roudsari & Wyatt, JAMIA, 2012); The flaws of policies requiring human oversight of government algorithms (Green, Computer Law & Security Review, 2022); EU AI Act, Article 14: Human Oversight
Yorumlar