The failures that make the news are ordinary mistakes nobody checked. Here's what a person in the loop is actually for, and how to tell when yours is just a signature.
AI failures that reach customers and courts are usually ordinary mistakes that skipped an ordinary check. The case for a human in the loop is three jobs: notice what’s off using context the model lacks, answer for the outcome, and decide what the work is for. A person who can’t see the source, say no or find the time isn’t a control. Score one real workflow with the six-question check and name an owner.
The AI failures that end up in front of a judge or a customer are rarely exotic. They’re ordinary mistakes: a court case that doesn’t exist, a refund rule nobody wrote, a footnote pointing at a paper nobody published. They left the building because nothing stood between the draft and the reader except the hope that it was fine.
Three cases show the pattern, and the details matter because each is smaller and more ordinary than the headline version. In 2023, two New York lawyers filed a brief in Mata v. Avianca that cited cases ChatGPT had produced and that didn’t exist. The judge’s sanctions order is worth reading for what it doesn’t say. It doesn’t say using AI was improper. The court wrote that there is nothing inherently improper about using a reliable AI tool for assistance, and that existing rules put a gatekeeping role on attorneys to make sure their filings are accurate.[1] Most of the order concerns what came after the other side couldn’t find the cases: the lawyers stood by the fake opinions, and two attorneys and their firm ended up with a $5,000 penalty.
The second is Air Canada. A customer asked the airline’s website chatbot about bereavement fares and was told he could claim the reduced rate after travelling, within 90 days. The airline’s own policy page said the opposite. A British Columbia tribunal ordered the airline to pay him the difference, and the full bill came to CA$812.02.[2] Small money, and a clear answer on who was responsible.
The third is a consulting deliverable. Deloitte Australia produced an independent review for the Australian government’s employment department, on a contract worth roughly US$290,000. An outside academic noticed references to papers that don’t exist and a quote attributed to a federal court judgment. The department said Deloitte confirmed some footnotes and references were incorrect, the corrected version disclosed that a generative AI tool was used, and Deloitte agreed to refund the final instalment of its fee.[3] The first alarm came from a reader outside the building.
| Case | What went out | The check that was missing | What it cost |
|---|---|---|---|
| Mata v. Avianca (US court filing) | A brief citing cases that didn’t exist | Someone opening each cited case before signing | A $5,000 penalty on two lawyers and their firm, plus letters to the judges whose names were attached to fake opinions |
| Moffatt v. Air Canada (customer chatbot) | An answer that contradicted the airline’s own policy page | Someone comparing what the bot says with what the policy says | CA$812.02 ordered, and a ruling that the airline answers for what its chatbot says |
| Deloitte and the Australian government (consulting deliverable) | A review with references to papers that don’t exist and a fabricated court quote | Someone following each footnote to a real source | A corrected version, an AI-use disclosure and an agreed partial refund; the errors were first raised by an outsider |
Facts from the court order, the tribunal decision and press coverage. Sources are listed at the end.
Read the missing-check column. None of those checks needed a specialist or a special tool. Each was a few minutes of work for someone who owned the output: open the case, put the policy page next to the bot’s answer, click the footnote. What was missing wasn’t capability. It was a person whose job that check was, with the time to do it. That’s the argument for a human in the loop, and it’s narrower and more useful than “AI makes mistakes”.
Specific jobs go undone unless a person is assigned to them.
Writing those jobs into a review process is the how-to side, and we’ve covered it in Human-in-the-Loop: How to Actually Write the Guidelines. This piece is the case for why, and for telling when the person in your loop is a real control and when they’re a signature.
A human in the loop usually gets pitched as a safety net, which is why it sounds like a tax on speed. A more useful way to see it: the person is doing three jobs the model can’t do for itself, and a loop that skips any of them is weaker than it looks.
Notice is the job people assume AI will absorb next, and it’s hard to automate because the evidence of a mistake usually sits outside the output. A finance analyst asks AI to draft the variance commentary for the month-end pack. The first three paragraphs are right. The fourth explains a cost overrun with a supplier price rise that didn’t happen, in the same level voice as the rest. She catches it only because she was on the supplier call. Nothing on the page gave it away. (For why models produce confident wrong answers in the first place, see why AI hallucinates.)
That matches what happened in a field experiment with consultants. The authors built one task to sit outside what the model does well, and the consultants who used AI were 19 percentage points less likely to get the right answer than those working without it.[4] What matters for you is that the experience gives little warning that you’ve crossed the line. The wrong answer arrives as polished as the right one, so past success isn’t a quality-control system.
AI sounds as sure when it’s wrong as when it’s right, so how confident it sounds is a poor guide to whether it’s right.
Answer is the job that can’t be handed to the model, and the Air Canada decision makes it concrete. The airline argued, in effect, that its chatbot was a separate entity responsible for its own actions. The tribunal called that “a remarkable submission,” said the chatbot is still part of the airline’s website, and held the airline responsible for all the information there, whether it sits on a static page or comes from a chatbot.[2] The same logic holds inside a company. A marketing manager approves landing-page copy at the end of the day, and one line says the product connects with every major CRM. It reads well and nobody opens the integrations page. When a prospect asks sales to confirm it, the claim is the company’s, and “the tool wrote it” isn’t an answer anyone is waiting to hear.
If you can’t name the one person who answers for an AI-assisted output, it isn’t ready to go out.
Decide is the quietest job. The model answers whatever it’s asked, and it won’t tell you the question was wrong, the audience was wrong, or that this message should have been a phone call. A person who decides what the work is for, and whether AI should touch it, is doing something no review of the finished output can do. That’s why “check it before it goes out” misses part of the job.
Here’s how the split looks where AI does a real share of the work.
| What AI does | What the human still owns | How it gets checked |
|---|---|---|
| Drafts the variance commentary from the ledger export and last month’s pack | Explaining why the numbers moved, which comes from conversations with budget owners, and deciding what leadership needs to hear | Each stated cause traced to a named person or document before the pack goes out |
| Drafts landing-page copy and product claims from the brief | Deciding what the company can truthfully promise, and owning every claim once it’s published | Each claim matched to the product team’s confirmed capabilities by someone other than the drafter |
| Summarises a supplier contract’s renewal and termination terms | Deciding what the terms mean for the business and whether to sign | The clauses the summary cites are read in the contract itself, not in the summary |
A Future Factors working view. The split changes with the task; a named person and a source to check against stay constant.
A common mistake is treating the person as the control. Once a human is “in the loop”, the risk gets marked as handled, and the question of whether that person can do the three jobs stops being asked. A hiring manager approves an AI-written summary of interview notes for a shortlist. The raw notes live in another tool, so approving the summary means reading the summary and asking whether it sounds right. It does. Sounding right wasn’t what was in doubt.
The pull of a visible AI answer is strong even for experts. In a lab experiment, radiologists read mammograms with AI suggestions, some of them wrong. Less experienced readers went from correct in almost 80% of cases to under 20% when the suggestion was wrong, and the more experienced group dropped too, though less sharply.[5] It’s a small group in a lab, and mammography isn’t your invoice review, so I wouldn’t carry the numbers over. I’d carry over the direction: a suggestion already on the screen shapes the check that follows, and experience helps without making anyone immune.
Three conditions usually turn a real check into a signature, and none of them looks like negligence from the inside:
Writing those conditions out as policy is the how-to half, and it’s the subject of our guidelines piece. The point here is narrower. A person in the loop and a check in the loop are different things, and only the second is protection.
When an AI-assisted output goes wrong, the first question from a customer, a tribunal or a board is rarely which model produced it. It’s who checked it, and the answer has to be a person. That’s the plain logic behind both court decisions above, and it’s now written into regulation too.
In the EU, Article 14 of the AI Act requires high-risk systems to be designed so people can oversee them effectively. The text spells out what that person must be able to do: understand the system’s limits, stay aware of the tendency to over-rely on its output, and decide not to use it or to disregard, override or reverse what it produces.[6] Tools used to recruit people or to manage and assess workers are on the high-risk list.[7] Article 22 of the GDPR runs on a related track. People have a right not to be subject to decisions based solely on automated processing where those decisions significantly affect them, and where such decisions are allowed, the safeguards include the right to obtain human intervention.[8] I’m not a lawyer, and I haven’t confirmed the current AI Act application dates, so treat this as a reason to ask counsel, not as advice.
Here’s the useful part. Requiring a person doesn’t make the person effective. A review of 41 policies that require human oversight of government algorithms concluded that people often can’t perform the oversight those policies assume, and that the requirement can give false reassurance while a flawed system stays in place.[9] The rule asks for a person. It can’t supply the source documents, the authority or the hour, and that part is yours to build.
The three jobs turn into questions you can score on one real workflow. Pick a task where AI does part of the work and a person approves the result, and score each line 0 (no), 1 (partly) or 2 (yes).
| # | Job | Question | What a 2 looks like |
|---|---|---|---|
| 1 | Notice | Is the source in view? | The reviewer can see what the output was built from, not only the output |
| 2 | Notice | Could they have done it themselves? | They know the subject well enough to feel a wrong answer |
| 3 | Answer | Is there one named owner? | One person answers for it to a customer, a court or a board |
| 4 | Answer | Can they stop it, and is there time? | They can hold the output without asking permission, and the workflow leaves room to do it |
| 5 | Decide | Was AI chosen for this task on purpose? | Someone decided AI belongs here and what it’s allowed to draft |
| 6 | Decide | Do they form a view first? | They decide what a right answer looks like before reading the AI’s |
Maximum 12. Score the task as it runs on a normal busy day, not as it’s described in a policy.
| Score | What it means | What to do |
|---|---|---|
| 10 to 12 | A real loop | Keep it, and re-score when the task or the tool changes |
| 7 to 9 | Partly decorative | Fix the lowest-scoring line before you scale the use |
| 0 to 6 | A signature, not a check | Fix the setup, narrow what AI does here, or take AI out of this task |
Bands are a Future Factors working guide based on our own judgment, not a benchmark.
Score the hiring manager from earlier. The source isn’t in view (0). She knows the role well (2). The owner is named, loosely (1). She could hold the shortlist, but there’s no slot for it (1). AI was chosen without deciding what it may leave out (1). She reads the summary before forming her own view (0). That’s a 5.
Her judgment isn’t the problem. The setup gives it nothing to work with.
A person in the loop counts as a control only if they can see the source, say no, and have the time to do both.
Two limits are worth stating. A low score doesn’t mean add another person: a second reviewer reading the same summary has the same problem as the first, so the fix is usually to change what the reviewer can see. And plenty of work needs no loop at all, like a rough outline only you will read. There, you’re the loop, and adding process costs more than it protects.
The three jobs land differently depending on what you own, because the context that makes someone good at Notice is different in each seat.
| Role | The context only this person holds | What a skipped check looks like |
|---|---|---|
| HR and people ops | The history behind a person or a policy, and what was said in the room | A summary that flattens a candidate’s situation into something tidy, approved because it reads well |
| Finance | Why the numbers moved, which usually lives in conversations with budget owners | A plausible explanation for a variance that nobody confirmed |
| Marketing | What the company can truthfully promise, and what customers have been told before | A claim in an ad or on a landing page that the product can’t back up |
| Operations and procurement | How the contract or the process actually runs day to day | A summary that gets the headline terms right and loses the carve-out in the termination clause |
| Leaders | Which decisions can’t be reversed, and who answers for them | Sign-off with no named owner below, so “who checked this?” has no answer |
A Future Factors working view, written from our own judgment about where the context sits in each role.
If you’re the one accountable for the loop, a common number to track is how many reviews were completed. That measures use, which is the easiest thing to check and the weakest evidence. Three questions get closer:
A loop where nothing was ever caught is either good news or a loop that isn’t looking, and only the third question tells you which.
Teams can start on this themselves: pick one workflow, score it, name the owner. Training with an experienced facilitator shortens the learning curve, though, because Notice sharpens on real work with feedback, and a team learns it faster than one enthusiastic person alone. In the terms we use at Future Factors, training builds the capability to spot what’s off, and workflow design gives it somewhere to be used: a source in view, a slot in the day, a named owner. Reinforcement, like a manager who now and then asks to see what was caught, turns it into behaviour, and adoption is what you see when it holds. Judgment is one of the skills that gains value as drafting gets cheaper, an argument we made in Human Skills in the AI Era.
If you need to argue for a real loop to a team or a boss, a list of horror stories carries only so far, because every audience has its own version of “that couldn’t happen here”. A pre-mortem works better, because the people in the room supply the failures. Paste this into your AI tool.
| I’m accountable for [OUTPUT, e.g. the monthly variance pack] and someone on my team uses AI to draft [WHICH PART]. Assume it goes out with a mistake that a person outside the team notices. List the five most plausible mistakes, given this kind of task and what the AI will and won’t have in front of it. For each one, tell me: who would notice first and how late; what it would cost us in time, money, trust or legal exposure, without inventing figures; and what a person who holds [CONTEXT ONLY OUR TEAM HAS] could catch in under five minutes. Then label each mistake Notice, Answer or Decide. Notice means someone with context could have caught it. Answer means someone would have to explain it to a customer or a boss. Decide means we shouldn’t have used AI for that part. Do not invent statistics or cases. If you’re unsure whether something is plausible for this task, say so. |
Treat the output as the start of a conversation. The model doesn’t know your team, so correct what it gets wrong first.
Run it with the people who do the task. They tend to recognise at least one of the five as a near miss, and that’s the case for a real loop, made in their words instead of yours.
Then, this week:
If the answer for that task is that a person belongs there, the next job is writing down what they check, and that’s what the guidelines piece is for. Start with one task and score it before you change anything.
A person reviews, approves or can override what an AI system produces before it has an effect. Two neighbouring terms get mixed up with it: human on the loop, where a person monitors the system and steps in when something looks wrong, and human out of the loop, where nobody does. Which fits depends on how fast a mistake spreads and how hard it is to undo.
It can eat some of the savings. A check that takes minutes against a task that took hours is usually a good trade. If the check takes as long as the work, the task probably needs a different design: a narrower job for the AI, or no AI. I haven’t found a trustworthy benchmark, so time it on your own task.
Sometimes, and not as a general rule. Whether you’re covered depends on the type of system and where you operate, so start by checking whether your tool is used for hiring or for managing workers, which is where the EU rules concentrate. Some client contracts and sector rules may also ask for human sign-off. Ask counsel before you rely on any of this.
Work where you’re the only audience and a mistake is cheap to fix: a rough outline, a brainstorm, notes only you will read. You’re already the check. That changes once the output reaches someone else or feeds a decision.
It can add a useful pass and shrink what the person has to read, and we cover the technique in using AI to evaluate AI. But the second model sounds as sure as the first, and neither can answer for the result. Use it to point a person at the likely problems, not to replace them.
Sources were read on October 14, 2026: the sanctions order in Mata v. Avianca, the tribunal decision in Moffatt v. Air Canada, Fortune’s coverage of the Deloitte case for the Australian government, the abstract of Dell’Acqua and colleagues’ consultant experiment, the RSNA summary of the Radiology mammography paper (we did not read the paper itself, and it covered a small group of readers in a lab), the abstract of Ben Green’s analysis of oversight policies, and the text of Article 14 of the EU AI Act, its Annex III and Article 22 of the GDPR. Harvard Business School’s summary page words the consultant result as 19%, while the authors’ own abstract says 19 percentage points; we used the abstract. The sources we read gave no Deloitte refund amount, so we did not state one. The three jobs, the loop reality check, its score bands and the role table are Future Factors working methods based on our own judgment, not research-validated models. The scenes in this guide are illustrations, not client stories. We did not use any figure for how often AI output contains errors or how much time review costs, because we could not confirm one against a primary source. This is not legal advice.
Weekly tips. Real examples. Practical help for busy professionals.
We care about your data, check out our privacy policy.