A person in the loop is a design to test, not a safety guarantee. Three ways it fails, and the test and fix for each.
A person in the loop is a design choice, not a safety guarantee. It tends to fail in three ways: the reviewer can’t see what matters, review is too late or too fast, and findings go nowhere. Test each with a signal you can measure this week. The diagnostic, scored check and log are Future Factors’ working tools.
Ask a team whether a person checks the AI’s work and the answer is usually yes. Ask what that person sees, when they see it and what happens to what they find, and the answers get vaguer. That gap is where human-in-the-loop setups tend to fail, and the failures look like success from outside: the step exists, the box gets ticked, and the same errors turn up next week.
It helps to treat a person in the loop as a design choice and not a safety guarantee. A design can be tested, and this one tends to break in one of three ways that have little to do with each other:
Each needs a different test and a different fix, which is why “add more review” often doesn’t change the outcome.
A fourth pattern gets a lot of attention, the reviewer who approves whatever the tool produces. That’s automation bias, and it comes down to who owns the decision, so it sits outside these three. For the case for having a person in the loop at all, see Why Human-in-the-Loop Matters; this guide assumes you have one and asks whether it works.
A content editor opens an AI-drafted article full of confident claims, none with a link. Checking means leaving the draft and searching for each claim, so the editor usually reads for flow instead, because flow is what the screen makes quickest to judge. A finance analyst gets variance commentary that explains a movement without showing which ledger query produced the figure.
That’s the first failure. The reviewer has a name on the review and little to review with. The output is the only thing that arrives; the inputs, the sources and the places the model was guessing stay behind in the tool. Fluent text is quick to approve and slow to check.
Showing more doesn’t fix it by itself. In a peer-reviewed experiment from Harvard researchers, adding explanations to an AI’s suggestions did not appear to reduce overreliance on wrong suggestions. Designs that made people decide first, wait, or ask for the suggestion on demand did reduce it on part of the task.[1] People liked those designs less, and the study used one low-stakes task with a simulated AI, so it’s a design hint, not a result to bank on. A reviewer needs the material in front of them and a step that makes them use it.
To detect it this week, ask the reviewer to trace five claims or figures from last week’s outputs to their sources. The signal is how many get traced in about a minute each. A second signal is the share of outputs that reach the reviewer with sources and inputs visible at all. If either is low, the reviewer has been judging style.
The fix is a review packet, not a longer checklist. Here is one filled in for two roles.
| Part of the packet | Content editor, AI draft | Finance analyst, AI-prepared commentary |
|---|---|---|
| Beside the output | A source for each factual claim; unsourced claims highlighted | The report or query behind each figure, with its period |
| Reviewer answers first | The claim the piece rests on, in one sentence, before reading on | The driver they expect behind the movement, before reading the commentary |
| Reviewer checks | The highest-consequence claims against their sources | Every figure that moves a total, tied to the ledger |
| Done when | Each flagged claim has a source or has been cut | Each figure ties, or is queried with its preparer |
A starting point to copy. The answer-first row borrows the commit-before-you-see idea from the study above.
A reviewer signs off only on what they can see: the output, its sources and its inputs.
For high-risk systems, the EU AI Act asks for something similar in legal terms. Article 14 says the people assigned oversight should be able to understand the system’s limits, interpret its output and stay aware of automation bias.[2] Ask counsel whether it applies to you; the design point holds either way. The 60-second check in the AI Trust Checklist pairs well with it.
A customer support lead reads the AI-drafted replies as a weekly quality sample, after the tickets have been answered. The sample helps coaching but not the customer who already got a wrong answer. A finance team reviews AI-written commentary the afternoon the pack goes out, when changing a sentence means reopening it.
The other version isn’t late, it’s crowded. An HR business partner has a queue of AI-summarised case notes, each needing a read against the original, and by the end of a sitting the review has become scroll and click.
Placement and volume are the two causes. The review went in at the end because that was the convenient place to add it, and every output gets the same review, so the few that carry risk are buried among the many that don’t.
A study of 112 primary care clinicians’ responses to decision-support alerts points at the volume problem, with a clear caveat: these were clinical alerts, not AI outputs. The likelihood of accepting a reminder fell by about 30 percent for each additional reminder per visit, and repeated, low-information alerts mattered more than how busy the clinician was overall.[3] Read it as a warning about queues full of items that don’t need a person.
Three signals tell you which version you have:
The fix has two parts. Move the gate ahead of the action, draft-and-hold instead of send-and-sample, wherever the tool allows it. Then size the queue to what one person can read at the speed the check takes, and route by consequence so the items that can hurt someone get the full read and the low-risk ones a spot check. Human-in-the-Loop: How to Actually Write the Guidelines covers matching review depth to consequence, so I won’t repeat it.
Review before the output acts, at a volume one person can actually read.
An editor who fixes the same unsupported claim in every draft is doing the job well and the system badly. The correction happens in the document, while the cause sits in the prompt, the template or the source material, and none of those change.
In a support team, wrong answers about a refund window get flagged in a shared channel and the flags scroll away. Two weeks later the assistant makes the same mistake, because the policy page it reads from is still the old one and nobody owned that page.
Reviewers rarely own the prompt or the source, findings get recorded as edits and not as causes, and fixing the output feels like the job being done. A loop is only a loop if something comes back around.
NIST’s AI Risk Management Framework treats this as part of the work, not an extra. It lists feedback processes for end users and affected communities to report problems and appeal outcomes (MEASURE 3.3), and post-deployment monitoring that captures and evaluates input, with measurable improvement activities built into system updates (MANAGE 4.1 and 4.2).[4] The framework is voluntary; the useful part is the shape, findings in and updates out.
To detect it, count the findings your reviewers made last month, then count how many led to a changed prompt, template, rule or source document. The second number divided by the first is a closure rate. The other signal is repeat rate, the share of this month’s findings that match a type you’ve already logged. A low closure rate with a high repeat rate means the loop is open.
The fix is a findings log that records the cause and the change, not just the correction. Four rows, filled in, show the standard:
| Finding | Type | Cause | Change made | Owner | Recheck |
|---|---|---|---|---|---|
| A claim in a draft had no source | Visibility | The draft prompt never asked for sources | Prompt now asks for a source per factual claim and marks gaps | Content lead | Next four drafts |
| Commentary named the wrong driver for a variance | Visibility | Commentary was generated without the cost-centre breakdown | Breakdown added to the inputs | Analyst who owns the pack | Next month-end |
| A reply quoted an old refund window | Source | The assistant read an outdated policy page | Old page retired; one policy page named as the source | Support lead | Weekly sample |
| A case summary left out a dated complaint | Prompt | The summary prompt said nothing about dates | Prompt now asks for a dated list of events | HR business partner | Next ten summaries |
Illustrative rows, not a real log. The columns are the point: type, cause, change, owner and recheck. Add a date column if your team wants one.
Then give it a rhythm. Once a week, for about fifteen minutes, the people who own the prompts, templates and source pages read the log, and each row closes with a change or a written reason for no change. Tell reviewers what changed too, or they stop reporting.
Every review finding ends in a change or a written reason.
The three failures show up differently depending on who the reviewer is, so the first fix differs too.
| Role | Failure to expect first | First fix |
|---|---|---|
| Content editor | 1: claims without sources | Each claim beside its source; the draft prompt returns a source per claim and marks gaps |
| Finance analyst | 1 and 2: figures without the query behind them, reviewed near the deadline | Tie figures before reading commentary; agree a cut-off ahead of release |
| HR business partner | 2: a heavy queue of summaries, sensitive items mixed with routine ones | Original notes beside each summary; full read for sensitive items, spot-check the rest |
| Customer support lead | 2 and 3: replies reviewed after sending; fixes that don’t reach the source page | Show the draft, the policy page and the customer’s message together; hold refunds, exceptions and complaints for a person; name an owner per source page |
Starting points to adapt. Begin with whichever failure the scored check in the next section rates lowest.
Behind each row sits a split of the work. Write it down for any workflow where review matters; for deciding what to hand to AI in the first place, see the decision matrix.
| Role | What AI does | What the human still owns | How it gets checked |
|---|---|---|---|
| Content editor | Drafts the piece and lists its factual claims | Which claims run; accuracy and voice | Editor traces the riskiest claims before publishing; lead reads a monthly sample |
| Finance analyst | Drafts commentary from the pulled figures | The numbers and the explanation of what drove them | Figures tied to the ledger first; a second person checks a sample |
| HR business partner | Summarises case notes | Any conclusion about a person’s pay, performance or case | Original notes read for flagged items; a sample of the rest |
| Customer support lead | Drafts replies from the policy pages | Exceptions, refunds and complaints | Held replies read before sending; weekly sample; errors traced to the source page |
A starting point to copy.
Reviewing AI output is a skill, and teams can start building it internally. Expert-led training shortens the learning curve, builds one standard across a team and ties practice to real work. It helps to keep the four parts apart. Training builds the skill of reviewing: knowing where AI output goes wrong in your kind of work. Workflow design puts the review at the right point with the right material (failures one and two). Reinforcement is the weekly habit that closes the loop (failure three). Adoption of a working review practice is what you see when those hold. Our corporate workshops work on a team’s real workflows; What Is an AI-Human System sets out the wider design.
Start with what you notice. A reviewer who can’t trace a claim has a different problem from one who clears a long queue in minutes, and the fix for one does little for the other. Match the symptom to the failure before you change anything.
| What you notice | Likely failure | Test it this week | First fix |
|---|---|---|---|
| Reviewers can’t say where a figure or claim came from | 1. Can’t see what matters | Can the reviewer trace five claims from last week to their sources in about a minute each? | Put sources and inputs beside the output; reviewer answers first |
| Errors are found after the email, report or payment has gone out | 2. Too late | Compare review time with effect time on ten recent outputs | Move the gate ahead of the action |
| A long queue is cleared in one sitting with almost no edits | 2. Too fast, too much | Plant one known error in a small sample | Cap the queue; route by risk |
| The same correction appears in the notes week after week | 3. No feedback | Count last month’s findings that changed a prompt, template, rule or source | Log each finding with an owner; close it with a change or a reason |
Future Factors’ own working tool, not a standard. Run it on one workflow at a time.
The tests then roll into one check a team can run in about an hour. Score each line 0, 1 or 2 for a single workflow.
| Question | 0 | 1 | 2 |
|---|---|---|---|
| 1. Visibility. Do outputs reach the reviewer with sources and inputs visible? | Few do | About half | Nearly all |
| 2. Tracing. Can the reviewer trace five claims to their sources? | Can’t trace them | Can trace some | Traces all, about a minute each |
| 3. Timing. Does review happen before the output acts? | After it acts | Before, for some | Before, for all |
| 4. Load. Can the reviewer finish the queue at the speed the real check takes? | No | Only by cutting corners | Yes |
| 5. Planted error. Was a known error caught in a small test sample? | Missed | Caught on some tries | Caught |
| 6. Feedback. Did last month’s findings lead to changes or written reasons? | None did | Some did | Nearly all did |
Add up to 12. Working-judgment bands, not validated cut-offs: 0 to 4 an open loop, mostly ceremony; 5 to 8 a leaky loop; 9 to 12 a working loop to keep testing. Questions 1 and 2 point to failure one, 3 to 5 to failure two, 6 to failure three. Fix the lowest group first.
To see whether anything is changing, measure use, then persistence, then impact. Counting outputs “reviewed” is a use measure at best.
| Question | Example for a review workflow |
|---|---|
| Use: are people doing it? | Share of reviewed outputs that had sources visible; findings logged this week |
| Persistence: still doing it a month later, unprompted? | Weekly triage still happens; the log keeps getting updated |
| Impact: is the work better or different? | Planted-error catch rate; repeat rate of findings; days from finding to change |
Write the before figure down first.
You need one workflow, one reviewer and two weeks.
| When | Do this | Done when |
|---|---|---|
| Days 1 and 2 | Pick one workflow and a named reviewer. Run the scored check | You have a score and a lowest-scoring group |
| Rest of week 1 | Trace five claims, compare review and effect times on ten outputs, plant one error | You have a tracing result, a timing gap and a catch result |
| Week 2 | Start the findings log. Hold one fifteen-minute triage with the prompt or source owner. Change one thing aimed at the lowest group | One finding closed with a change; an owner and recheck date on each row |
| End of week 2 | Score the check again | You know whether the change moved the score |
Copy into your planner.
Some findings will say review is the wrong control for that workflow. If the reviewer can’t be shown what they need, or the volume can’t come down, try a narrower use of AI, an automated check, or none. For the wider view, see Managing the Responsible Use of AI.
This week, take five claims from last week’s reviewed outputs and ask the reviewer where each came from.
Because “reviewed” only says someone looked, not what they could see, when they saw it or what changed afterwards. Reviewing everything at the same depth can itself be a cause, because low-risk items crowd out the few that carry risk.
Run three small tests: trace five claims to their sources, plant one known error in a small sample, and count the share of last month’s findings that led to a change. A few planted-error tries are a rough signal, so repeat them.
There isn’t a universal number. Time the careful check on a few real items, then divide the time a reviewer actually has by that figure. If the queue is bigger, route on consequence and trim what doesn’t need a person.
Log each with its type and cause, give the fix to whoever owns the prompt, template, rule or source page, and set a recheck date. Close every row with a change or a written reason, then tell the reviewer what changed.
When the reviewer can’t be shown what matters and you can’t fix that, when volume is too high for a person and the risk is low enough for sampling or an automated check, or when the person would just redo the whole task. Some legal settings require a person to decide, so ask counsel.
I read the sources on November 2, 2026: the AI Act Service Desk’s reproduction of Article 14 (not the official EUR-Lex text), Buçinca, Malaya and Gajos (2021) on arXiv, NIST’s AI Risk Management Framework 1.0, and the abstract of Ancker and colleagues (2017) via Europe PMC, since the publisher pages blocked automated access. The Buçinca study used one low-stakes task and a simulated AI, and the Ancker study looked at clinical alerts, so both are design hints. The failures, diagnostic, scored check and log are Future Factors’ working tools and judgment, not standards, and nothing here is legal advice. I didn’t test any product hands-on, and the scenes are illustrations, not client stories.