Designing Human-in-the-Loop Approvals for AI Agents

Teams usually arrive at this from one of two extremes. Either the agent asks permission for everything, and people start clicking "approve" without reading, or it asks for nothing, and the first wrong email to a customer ends the project. Neither is a design. This post covers the patterns I use to place approvals where they earn their cost.

What does human in the loop mean for an AI agent?

For an agent, human in the loop means specific actions cannot execute until a person has seen and decided on them. It is narrower than "a human is involved somewhere". The agent can read, search and draft freely; the gate sits in front of actions with consequences.

The idea is built into the standards and frameworks agents run on. The Model Context Protocol's tools specification says "there SHOULD always be a human in the loop with the ability to deny tool invocations", and that applications should present confirmation prompts. Anthropic's Building effective agents describes agents that "can then pause for human feedback at checkpoints or when encountering blockers."

Regulation points the same way for high-risk systems. Article 14 of the EU AI Act, as published on the AI Act Explorer, requires that people overseeing high-risk AI can decide not to use it, "disregard, override or reverse" its output, and stop it through a "'stop' button or a similar procedure". It also names "automation bias", the tendency to over-rely on output, as something overseers must stay aware of. I covered the wider obligations in the EU AI Act deadline post.

Which agent actions need human approval?

Actions need approval when a mistake would be expensive, visible to outsiders, or hard to reverse. I sort every tool an agent can call into tiers before writing any approval code.

The tier belongs to the action and its arguments, not just the tool. A refund tool might run automatically below a small amount, need one approver above it and two above a higher one. Frameworks support this directly: the OpenAI Agents SDK lets needsApproval be an async function that receives the tool's parsed arguments and decides per call, according to its human-in-the-loop guide.

What are the main approval patterns?

There are five patterns I use, often several in one agent.

• Approve before execution. The agent proposes; the run pauses; a person approves or rejects; the run resumes. This is the default for external actions.
• Edit and approve. The reviewer changes the proposal, such as fixing the tone of an email or the amount of a refund, before approving. It turns a rejection into a correction and keeps the work moving.
• Stage gates. In a multi-step pipeline, a person approves each stage's output before the next stage starts. My AgentOps project works this way: a request in Slack becomes a spec, then tickets, then a pull request and a preview, and nothing moves to the next stage without a person approving it, so an early mistake does not fan out into tickets and code.
• Review after the fact. For low-risk reversible actions, let them run and review a sample. It catches drift without slowing anyone down.
• Escalate on blockers. When the agent is stuck, missing information or outside its scope, it stops and asks rather than guessing.

Should approvals be based on the agent's confidence?

Use confidence thresholds only when the confidence comes from something you can check, not from the model rating itself. A model asked "how sure are you?" will produce a number, but that number is another generated output, not a measurement.

Signals I trust more for routing:

• Validation results. Did the output pass the schema, the business rules, the totals check?
• Retrieval evidence. Was there a source document supporting the action, and does the agent cite it?
• Business thresholds. Amount, number of recipients, customer tier, data sensitivity.
• Novelty. Has this kind of action, for this kind of record, been approved before?
• Agreement. Did two independent checks, such as a rule and a second model, agree?

Route to a human when any trusted signal is weak. Then track how often reviewers overturn auto-approved categories; if they never do, the threshold can loosen, and if they often do, it should tighten. My post on giving AI agents a way to check their work covers building those checks.

How do you implement a pause that survives hours or days?

Save the agent's full run state to durable storage when it pauses, and resume from that state when the decision arrives. Approvals rarely come back in seconds, so the pause cannot live in memory.

The main frameworks handle this the same way:

• LangGraph pauses with interrupt() inside a node and resumes when you call the graph again with Command(resume=...) and the same thread_id. Its interrupts documentation recommends a "durable checkpointer in production", and warns that the node re-runs from its beginning on resume, so code before interrupt() executes again. Keep side effects after the interrupt, or make them idempotent.
• OpenAI Agents SDK returns pending approvals in result.interruptions; you call result.state.approve(interruption) or reject, then resume with runner.run(agent, state). State serialises with result.state.toString() and restores with RunState.fromString().

That guide adds a warning worth repeating: RunState.fromString() "does not authenticate the snapshot or the person submitting it." For a web approval screen, it recommends keeping the full state on the server, sending the browser only display details and an opaque decision ID, authenticating the reviewer, and consuming each pending request once. In other words, the approval endpoint is a security boundary, and it needs the same care as a payments endpoint.

What should the reviewer see, and what should the audit trail record?

The reviewer should see exactly what will happen, why, and on what evidence, in a form they can judge in under a minute. Approval screens that show a vague summary teach people to click "approve" without reading, which is automation bias by design.

On the approval screen:

• The action and its exact arguments: the recipient, the amount, the record ID.
• A preview or diff of the change, not a description of it.
• The evidence the agent used and its stated reason.
• Approve, edit and reject, with a short reason required for rejection.
• What happens if nobody responds, such as expiry after a set time.

In the audit trail:

• The proposal as generated, and the action as executed, if different.
• Who decided, when, and with what reason.
• The model, prompt version and tool versions in use.
• Links to the conversation or run that produced the proposal.

Rejections and edits are the best training data you have. Each one shows where the agent's judgement and yours differ; add them to your evaluation set, as I describe in hardening an AI feature for production.

How do you avoid approval fatigue?

Approve less, but better. If reviewers see dozens of trivial approvals a day, they stop reading them and the gate becomes theatre. Move actions down a tier once their overturn rate is consistently low, batch similar approvals into one screen, and keep the high-stakes ones rare enough that each gets real attention.

Work with me

I design and build agents with approval steps, risk tiers and audit trails, so teams can let them act without losing control. See AI agent development and workflow automation, or look at my work and book a call.