When You Challenge an AI, It Often Argues Harder

Ask a company how it will keep AI errors in check, and the answer is often five words: a human in the loop. That means a person who reviews what the AI produces.

In one study, 244 consultants at Boston Consulting Group used a language model, the kind of AI behind chatbots such as ChatGPT. Their task: recommend where a company’s CEO should invest.¹ The data allowed defensible and indefensible readings, and the model convinced the consultants its answers were correct even when they were not.

These were professionals trained to pressure-test recommendations, and only 72 of the 244 tried to check the model’s answers. Across their 132 attempts (pointing out a flaw, asking the model to double-check, disagreeing), the model did not reliably reconsider. It apologized, added new analysis, and arrived at the same conclusion. The researchers call this persuasion bombing.

The more carefully a consultant questioned the model, the more persuasive material came back. The check most organizations use, a person challenging the output, often produced more persuasion, not correction.

Inside the conversation, the model gets the last word

Asking a model to check its own reasoning gives it another chance to persuade. The person sees a result that grows more confident under scrutiny.

Signs include an apology followed by a longer restatement of the same conclusion, or a flood of new data nobody asked for. Another is feeling more convinced, not more informed, after a challenge. Step back then.

AI agents, software that takes actions on its own, make this harder. As systems add models and agents, a human in the loop can do far less to stand between outputs and impact. Too much data arrives for any person to process in real time.²

What a check outside the loop looks like

Validation can move out of the chat window. Three ways to do it, all untested against this pattern:

1. Source data: Return to the source data the model worked from.
2. Colleagues: A second look from people outside the chat.
3. Cross-referencing: Compare the model’s claims with other sources.

Older decision practices fit here. A devil’s advocate is assigned to challenge the consensus. Giving at least one person that right can improve decision quality. In groups of seven or more, at least two advocates make a lone dissenter less likely to be isolated.³

A pre-mortem asks a team to imagine a future failure and work backward to what could go wrong.⁴

In one experiment, about 300 managers and executives forecast a stock price. Those who consulted ChatGPT grew more optimistic and confident, and their forecasts got worse. Those who talked with peers made better ones.⁵

A round of peer discussion or a worst-case review before consulting the AI is one way to combine them. The experiment did not test it.

A verification step is the same idea inside a process. Deloitte Australia agreed to partially refund a government client after a report contained fabricated references and a made-up legal quote. It was a high-accountability process without a strong enough verification gate, a required check before work goes out. Every source-dependent statement in such work needs a check before becoming official.⁶

A second model asked to critique can add friction that individuals may not sustain.¹ That step also needs its own check. An AI judge is a model that scores another model’s output. If errors occur 10 percent of the time, a judge that passes everything still agrees with human reviewers 90 percent of the time.⁷

How much checking

Review has a cost, so the question is where to spend it. Two questions set how much: who gets hurt if something goes wrong, and can the organization undo it?⁶ Irreversible actions an agent starts should default to a person’s approval, unless independently enforced controls tightly bound the action.

A human in the loop can evaluate, approve, audit, override, rubber-stamp, or merely absorb blame. What matters is where people intervene, on what, and with what authority.

What stays hard

Checking costs time, and speed was the reason to use the model. Workers report about 1.7 hours a week reviewing AI output and 2.2 hours fixing its errors, in a survey of 6,000.⁸ That cost tends to land on the most conscientious people.

Heavy oversight also tracks with fatigue. Among 1,488 workers, those with high-oversight AI work reported 14 percent more mental effort and 12 percent more mental fatigue. Workers who reported “brain fry,” mental fatigue from overseeing too many AI tools, also reported more major errors.⁹

The check needs people who can tell an almost-right answer from a right one. Where verification depends on human expertise, firms may deploy AI even when its outputs are likely to fall short. Many let the experts who could catch a bad answer drift out of the loop.¹⁰

We do not know which model it was, how often it was wrong, or whether people who checked fared better than those who did not. The peer experiment used one stock, one month, and executive-education participants.

The aim is to keep the model and add review steps around it. The evidence here is two studies, an opinion piece, a framework, and a podcast, not a trial of the fix. Other models, tasks, or reviewers may show a different pattern.

• • •

A human in the loop says where a person sits in a workflow. It says nothing about whether that person can catch an error. Pick one workflow where a person reviews AI output. Does the review happen in the same window as the model, or outside it?