Human-in-the-Loop: How to Actually Write the Guidelines

Why "a human should review this" isn't a guideline, and what happens when that's all a team has

Say a marketing ops lead is at her desk on a Thursday afternoon, three hours from a scheduled send. The campaign email came back from the AI drafting tool twenty minutes ago. The company’s AI policy says one thing about this exact moment: “AI-generated content must be reviewed by a human before publishing.” She opens the draft, scrolls it once, sees nothing that jumps out, and clicks approve. That’s the review. That’s the entire instruction the policy gave her, and she followed it exactly.

Nothing in that sentence told her what “reviewed” actually means. Read it for typos? Check every number against the source pricing sheet? Confirm the send segment matches what was promised in the last customer email? The policy doesn’t say, so under a deadline, review collapses into whatever takes the least time and still lets her click the button. That’s not a lapse in judgment on her part. It’s what happens to any instruction that names an action without naming a standard.

It’s worth naming this failure mode precisely, because it doesn’t look like a failure while it’s happening. The guideline gets followed. Someone genuinely did review the output. Nobody can point to a rule that was broken. And the email still goes out to the wrong customer segment, because “review” was never defined as anything more specific than “look at it before you send it.”

A guideline that only says review is required, and stops there, functions less like a policy and more like a wish. Everyone nods along in the meeting where it gets approved, and nobody actually knows what to do differently the next time a draft lands on their desk at 3:45pm. A working guideline has to answer three questions for every task, and leave any one of them unanswered and you’ve written something that reads well in a policy document and produces a rubber stamp the first time someone is busy, which, on any real team, is most of the time:

• Who reviews it. A named role, not “a human” or “the team.”
• How deeply. A skim, a checklist against source data, or a two-person sign-off, decided in advance, not improvised under deadline.
• What they’re specifically checking for. The exact things to verify and where the source of truth for each one actually lives.

The fix isn’t more review across the board. Reviewing every AI draft with the same intensity, an internal meeting recap and a customer-facing price change alike, burns out reviewers and teaches them that most of it doesn’t matter. The fix is matching scrutiny to what’s actually at stake if the task is wrong and how easily that mistake can be undone.

Matching review depth to consequence, instead of reviewing everything the same way

Two things decide how much review a task needs: how bad it is if the output is wrong, and how easily that mistake gets caught and undone before it causes real damage. A typo in an internal Slack summary is low-consequence and fixed in a reply. A wrong number in a price-change email to ten thousand customers is high-consequence and, once sent, effectively permanent. Those two tasks should never get the same review.

Most teams skip this step and default to one of two extremes. Either everything gets the same light touch, which holds up fine until the one task that actually needed scrutiny slips through with it. Or everything gets the same heavy review, which means a director spends twenty minutes reading a Monday competitor-research note that nobody downstream will act on anyway. Neither extreme is really a policy. It’s a guess dressed up as consistency, and our broader look at where AI governance tends to break down at mid-size companies covers a version of the same problem, the gap between adoption moving fast and oversight staying still.

Here’s a framework that sorts tasks into four tiers by risk and reversibility, and assigns each one a review depth and a reviewer seniority level that actually fits. This is a different question from whether a tool gets approved for use in the first place, which is its own decision covered in our guide to AI governance without a legal department. That’s about who signs off on adopting a tool. This is about how deeply you review what an already-approved tool actually produces, task by task. Adapt the specific examples below to your own work; keep the logic connecting the columns.

Naming the tiers is the easy half. The harder half is writing down, for one real task, exactly what “line-by-line review” means in practice, because that’s the part most guidelines skip, and it’s the part that quietly turns Tier 3 review into Tier 1 review whenever the reviewer is behind.

Writing the actual guideline: who reviews, what they're checking for, and what happens when they push back

Say a subscription company is raising prices for existing customers, effective in sixty days, and the AI drafting tool has produced the announcement email that will go to the full customer list. This sits squarely in Tier 3: high stakes, hard to reverse, no do-over once it lands in ten thousand inboxes. The guideline says a senior reviewer checks it line by line. Here’s what that actually has to mean, written down as something the reviewer can run in five minutes instead of a vague instruction to “read it carefully.”

• Numbers match the source, not the draft. The new price, the effective date, and any grandfather-clause exceptions get checked against the actual pricing sheet, not against what the AI wrote, because a confident-sounding wrong number is the easiest thing to miss on a skim.
• Segment logic is correct. Customers on legacy plans, annual contracts, or promotional pricing usually need different wording. This is the row people skip under time pressure, and it’s usually where the real damage happens: one wrong sentence sent to the wrong dollar segment.
• Nothing is promised that isn’t confirmed. AI drafts sometimes soften a price increase with a specific commitment, a discount code, a locked-in renewal rate, that nobody on the pricing team actually approved.
• Tone survives a bad-faith read. The reviewer reads it once as a customer who’s already annoyed about the price going up, not as a marketer who’s pleased with how the copy turned out.
• The send list matches the wording. If the email says “as a loyal customer of three years,” the audience filter actually has to select customers of three-plus years, not the full list.

Notice what this checklist is not. It doesn’t say “review the email.” Every line names a specific thing to check and where to check it against. A reviewer running through this in five minutes catches more than someone reading the whole thing twice with no checklist at all, because a checklist points attention at the places mistakes actually hide instead of trusting that a careful read happens to land on them anyway.

Then there’s the part most guidelines leave blank: what happens when the reviewer actually catches something. Say the marketing director running this checklist finds that the segment logic is wrong, the legacy-plan discount language is going to everyone, not just legacy customers. The guideline needs to say, in advance, that the send is blocked, not quietly adjusted, and the fix goes back to whoever prompted the draft with the specific line that’s wrong, not a general “this needs work.” If the deadline is genuinely at risk because of the fix, the guideline needs to name who has the authority to move the send date, rather than let time pressure make that call by default.

This exact checklist won’t fit every task, and it isn’t meant to. What transfers is the shape: five to seven points, each naming a specific thing to check and where the source of truth for it actually lives, plus a defined answer for what happens when something gets caught. If a line in your version just says “check for accuracy,” you’ve written the same empty instruction you started with, only now it has a bullet in front of it. The same logic that makes delegating a task to a person work, a clear brief plus a defined way to check the output, is exactly what’s covered in our guide to delegating work to AI without losing control, and it holds here for the same reason: vague instructions produce vague results whether the thing doing the work is a person or a model.

Where human-in-the-loop quietly breaks down in practice, even with a guideline on paper

A written checklist solves the rubber-stamp problem on day one. It doesn’t solve it forever. Three patterns tend to show up months into a well-designed guideline, and none look like the guideline failing. They look like the guideline being followed by someone who’s quietly stopped using it.

The first is habituation. A reviewer who has checked two hundred price-change emails and found real problems in three starts, reasonably, expecting the two hundred and first to be fine too. The checklist still gets run, but faster, with less attention on the rows that have never caught anything. This is exactly how a Tier 3 review quietly becomes a Tier 1 review without anyone deciding to lower the bar. The fix isn’t a reminder about vigilance. It’s rotating who reviews a given task type every quarter or so, so fresh attention resets what “normal” looks like, and logging every catch so a lead can see when a task’s catch rate has sat at zero for long enough to be worth checking rather than trusting.

The second is tier drift. A task that started as an internal draft, reviewed lightly by whoever asked for it, quietly becomes customer-facing months later because someone found a faster way to publish it, and nobody moved it up a tier. The guideline still technically applies. It’s just applying the wrong tier’s rules to a task that outgrew them. Worth a standing rule: whenever a task’s audience or reach changes, whoever owns the guideline re-checks its tier before the old review level keeps running on autopilot.

The third, and the one guidelines usually handle worst, is disagreement. A reviewer flags something. The person who drafted it, or the manager who wants it out the door, disagrees it’s a real problem. Without a defined path, this gets resolved by whoever’s more senior or more insistent, or by the deadline simply arriving and forcing a default answer. That’s pressure making the call, not review.

Step five is the one worth protecting hardest, because it’s the one that gets quietly dropped first. A deadline is not a reason to skip a Tier 3 review, and a guideline that allows that exception even once has taught everyone downstream that the tier system is optional under pressure, which is exactly the pressure it was built for.

Reviewing and updating the guideline itself as the work and the tools change

A human-in-the-loop guideline is itself a piece of work that needs its own review, and the natural instinct is to measure whether it’s working by counting how often it got followed. That’s a start, not an answer. A reviewer clicking approve in four seconds is “following” the guideline in the narrowest sense, and it tells you nothing about whether the review caught anything.

Three questions get closer to whether a guideline is actually doing its job, not just existing. Use: are reviewers actually running the checklist, or approving from memory of what usually happens? Persistence: a month after the guideline shipped, is it still being followed the same way, or has it quietly eroded back into a skim? Impact: when a reviewer does catch something, does it actually change what goes out, or does the catch get logged and overridden anyway? A guideline that scores well on use and badly on impact isn’t a review process. It’s paperwork sitting next to a decision that gets made without it.

A clear signal worth tracking is the catch rate over time, broken out by tier. If Tier 3 reviews haven’t caught anything real in three months, that’s either genuinely good news, the drafting process upstream has gotten more reliable, or it’s the habituation problem from the last section wearing a disguise. The clearest way to tell the difference is to look directly at the record, which is why the log from the disagreement flow above earns its keep here too: it’s the same data that tells you whether review is still doing anything.

None of this needs a governance committee to get started. Pick one task your team already runs through an AI tool regularly, the one that would actually hurt if it went wrong. Write down which tier it belongs in, name the specific person who reviews it, and list the specific things they’re actually checking for, not “review it.” Run it for a month. Then look at whether anything got caught, and whether the catch changed what went out. That’s the whole guideline, for one task, and it’s the version worth having instead of the two-line policy that says review is required and leaves everyone to guess what that means.

Hina Mian, Co-Founder of Future Factors AI

Hina is a marketing strategist with over a decade of hands-on campaign experience across B2B and consumer brands. She writes about using AI to run leaner, sharper marketing without losing the human touch. Future Factors helps professionals and teams build practical AI capability through role-based training, workflow design, and hands-on adoption.

More about Hina →