Security Objective
Human-in-the-loop (HITL) is the control that places an informed human decision in front of selected consequential or ambiguous agent actions before they take effect. It answers three questions that no other control on this site owns: when a human decision is required, what evidence the reviewer must see to decide well, and how that decision is bound to the exact operation that will execute.
HITL is an additional gate for selected actions. It is not the base permission model. Deterministic agent authorization stays authoritative for whether a concrete action may execute at all, least privilege bounds how much authority exists, and information flow control decides whether data may move from a source to a sink. Human approval sits on top of those controls and never replaces them.
Ownership Boundaries
| Control | Question it answers | Not its job |
|---|---|---|
| Least privilege | How much authority, reach and lifetime exists at all | Deciding one concrete request |
| Agent authorization | Whether this fully resolved action may execute under deterministic policy | Judging context a policy cannot express |
| Information flow control | Whether data with this provenance and classification may reach this sink | Approving business intent |
| Human-in-the-loop | Whether a selected consequential action should proceed, after an informed human decision | Catching ordinary unauthorized actions or serving as the permission model |
If HITL is the first place an ordinary unauthorized action gets caught, authorization is missing. Every approval request should already be a request that deterministic policy would permit on its own, which HITL then subjects to an extra judgement because of risk.
Threat Model
The threats this control addresses are the ones every agent attack article on this site converges on: an agent influenced through indirect content or tool output proposes an action that is within its authority but wrong for the task. A second class of threats targets the approval step itself.
The action type and principal are allowed, but the exact recipient, target or parameters serve an injected objective.
Model-authored summaries, rationales or UI text describe the action more benignly than it is.
An approval for one action is replayed for a changed, retried or different action.
High volume or low signal trains reviewers to approve without reading.
Notifications or messages that imitate a genuine approval request.
The initiating user, or someone without authority for the risk class, approves their own consequential action.
Trust Model
Authoritative metadata and variable content stay separate all the way through the review. The reviewer's decision is an authenticated system event. The proposal it applies to remains variable content.
An authenticated initiating user does not turn their task content into system content. The model's plan, proposal and rationale are variable. Identity, policy decision, risk tier and provenance labels come from authoritative stores. The human decision is recorded as an authenticated event, but approving a message body, a diff or a summary does not relabel that payload as trusted. Execution stays behind deterministic enforcement.
Approval Is Not Authorization
Authorization runs before the approval request and runs again after approval, at execution time. If policy denies, the action does not execute, no matter who approved it. Approval can therefore narrow an outcome but never broaden it.
Approval Is Not Capability
Clicking approve must not mint a credential, add a scope or unlock a tool the agent did not already hold under least privilege. If an action needs authority the agent lacks, that is a provisioning question with its own review, not something an approval dialog resolves. The execution step uses the same bounded identity and credentials it would have used without the approval.
Who Decides That Approval Is Required
The model must not decide whether an action needs approval. A model that can say "this is low risk, no review needed" gives an injected instruction the same power. The requirement is resolved by deterministic policy outside the model, from the resolved action, its parameters, the principal, the data classification and relevant state.
Risk Tiering Criteria
There is no universal list of actions that require approval. Each deployment defines its own tiers. Categories that are frequently placed in a higher tier include the following, as examples only:
Deletion, overwrite, revocation without restore path.
Sending messages, publishing, posting to third parties.
Payments, refunds, purchases, pricing changes above a threshold.
Creating accounts, changing roles, granting access.
Releasing changes, running code outside a sandbox.
Exporting or sharing classified or personal data.
Microsoft's Agent Framework safety guidance, as vendor guidance, similarly recommends gating high-risk tools and names side effects and data sensitivity among the considerations. Tiers should be driven by consequence and reversibility, not by tool name alone.
Exact-Action Binding
An approval is only meaningful if it applies to the fully resolved operation. Approving the intent "send the report" does not approve a specific recipient, attachment and body.
What the approval binds to
- Action and tool
- Resource and target
- Destination or recipient, resolved
- Parameters, including amount and scope where relevant
- Data classification and provenance of the payload
- Expected side effects and reversibility
- Policy version and risk tier applied
- Material state the decision depended on, such as current balance, record version or recipient domain
How that binding is represented varies: a canonical action digest, a server-held action record or a versioned request object are all possible. The property matters more than the mechanism: the executor must be able to verify that what it runs is what was approved.
Material Change, TOCTOU and Re-Approval
Between approval and execution the world can change. A later planning step can rewrite a parameter, a recipient alias can resolve differently, a file can be replaced, or a balance can drop. This is a time-of-check to time-of-use (TOCTOU) problem. Any material change before execution invalidates the approval and requires re-evaluation by policy and, if still required, a new human decision.
What counts as material is part of the policy, not something the model decides. A useful default: any change to a bound field is material, and any change to state the decision depended on is material.
Informed Reviewer Context
A reviewer can only be as good as the evidence in front of them. The review surface should show two clearly separated groups.
| Group | Contents | Trust |
|---|---|---|
| Authoritative metadata | Initiating principal, agent identity, policy decision, risk tier, data classification and provenance labels, policy version | System, from authoritative stores |
| Proposal content | Action, parameters, resolved resource and destination, expected side effects, reversibility, diff or preview | Variable, shown exactly as it will execute |
| Model rationale | Why the agent says it wants to do this | Variable, an untrusted explanation, not evidence |
The task objective as stated by the initiating principal should also be visible, so the reviewer can compare what was asked with what is proposed.
Preview and Diff
Where it is meaningful, show the effect rather than a description of it: the exact message and recipients, a field-level diff of a record, the files a deletion will remove. A rendered preview must be generated from the bound action itself, not from a model-written summary of it. Where a preview cannot represent the effect faithfully, for example a large batch operation, say so and show scope counts and samples instead of an implied completeness.
Reviewer Identity, Eligibility and Separation of Duties
Approvals should come from an authenticated reviewer whose eligibility for the risk class is resolved by policy. For some classes, such as financial transfers or permission grants, separation of duties is appropriate: the initiating principal should not be the only approver of their own consequential action. Delegated approval authority should be explicit, bounded and recorded, not inferred from job title or chat context.
Decision Options Beyond Approve and Deny
The exact bound action may proceed to revalidation and execution.
The action does not execute; the agent receives a denial, not a hint to rephrase.
The reviewer edits parameters. This creates a new proposal that goes through authorization and binding again.
The decision moves to a reviewer with higher eligibility or domain knowledge.
Approval Fatigue, Rubber-Stamping and Batching
An approval step that fires on everything trains people to click through it. Fatigue is a design outcome, not a user failing. Keep review volume proportional to risk, remove approvals for actions that deterministic controls already handle safely, and avoid batching unrelated consequential actions into one decision. Where batching is unavoidable, a batch approval should still bind to an enumerated, fixed set of actions.
Deceptive Review Surfaces
Review-surface risks
- Model-authored titles, summaries or button text that frame the action as routine
- Summaries that omit a recipient, attachment or side effect present in the bound action
- Dark patterns such as pre-selected approval or hidden reject options
- Spoofed notifications or messages imitating a genuine approval request
- Rationale text styled like system text
Mitigations are structural: render system fields from authoritative data in fixed positions, visually separate variable content, deliver approval requests through an authenticated channel, and never let the model control the controls.
Multi-Step and Multi-Agent Workflows
Approving action A does not approve B or C. Approving a delegation from agent A to agent B does not approve B delegating onward to C. Each materially distinct action or handoff is evaluated separately against policy, and each one that the risk tier covers gets its own gate. The multi-agent and AI agent articles describe why chains amplify influence; HITL must not become the step that collapses a chain into one blanket yes.
Retries, Loops and Approval Lifetime
Conceptually an approval should be single-use, bound to one task and short-lived. A retry after a failure, a loop iteration or a resumed workflow must not silently replay an earlier approval. Whether this is enforced by one-time identifiers, expiry, or server-side state is an implementation choice; the requirement is that the executor can tell a fresh approval from a stale one.
Timeouts and Non-Response
For consequential actions, no answer means no. A request that times out, is abandoned or cannot be delivered should leave the action unexecuted and the task in a clear state, never fall through to execution by default.
Cancellation and Revocation Before Execution
An approved action may still need to stop. A reviewer, an operator or an automated signal should be able to cancel a pending approved action before it executes, for example when new risk is discovered or relevant state changes. Cancellation should be recorded like any other decision.
Break-Glass and Avoiding a Bottleneck
Emergency paths exist, and they should be separately governed: distinct eligibility, narrow scope, time limits and full audit. They are not a way around a policy deny through the normal approval flow. At the same time, HITL should not become a universal bottleneck. If everything needs approval, the control degrades into ritual. Invest first in authorization, least privilege and information flow so that fewer actions need a human at all.
Human Review Is Fallible
People miss details, trust fluent explanations and approve under time pressure. That is why deterministic controls remain authoritative before and after the human decision. HITL adds judgement where policy cannot express context; it does not make the system safe on its own.
Audit Lineage
Authenticated reviewer identity and eligibility basis.
The bound action identifier or digest, where available.
Request, decision and execution times.
Policy version and risk tier.
Relevant state before execution and the resulting state.
Execution result, or the reason for rejection, cancellation or invalidation.
Worked Synthetic Scenario
All data here is synthetic. A support operations agent processes a ticket whose attached text contains an injected instruction. It proposes emailing a customer data export to an external address. Deterministic authorization allows the send-mail action type for this principal in principle, but policy classifies an external send with a personal-data attachment as requiring human approval.
{ action: "send_mail", to: "export-review@example.net", attachment: "customers-q3.csv", body: "<model-written text>" }Action type and principal permitted; recipient domain is external; attachment classified as personal data. Policy result: allowed only with approval, risk tier high.
Authoritative fields: initiating principal, ticket ID, risk tier, data class 'personal', policy version. Separate variable fields: exact recipient, attachment name and row count, body, and the model rationale, marked as untrusted explanation.
The reviewer sees that the ticket asked for a status update, not an export, and rejects. Alternatively they modify the action to a status reply to the ticket's verified customer address without attachment, which becomes a new proposal.
Rejected: nothing is sent. Modified: the new proposal passes authorization at a lower tier and is sent. The export never leaves.
Benign case: an agent proposes a refund of a synthetic 40 units to the verified original payment method on a verified order. Policy allows it with approval above a threshold. The reviewer approves the bound action. Before execution the executor confirms the order state and amount are unchanged and authorization still permits it, then executes under the agent's normal scoped credentials.
| Gate | Rejected case | Benign case |
|---|---|---|
| Authorization | Allowed the type, required approval | Allowed with approval, re-checked at execution |
| Human-in-the-loop | Rejected or modified | Approved the exact action |
| Binding / revalidation | Not reached | Confirmed unchanged state |
| Least privilege | Bounded the credentials used | Bounded the credentials used |
Testing and Metrics
Measure each property on its own, with an explicit denominator and no composite score.
| Measure | Numerator | Denominator |
|---|---|---|
| Approval-policy coverage | Action classes with a defined approval rule | Action classes that policy designates as requiring HITL |
| Required-approval enforcement coverage | Required-approval actions that could not execute without a valid approval | Required-approval actions tested |
| High-risk proposal containment rate | Unsafe proposals rejected, modified or contained before effect | Unsafe proposals in controlled tests |
| Unsafe approval rate | Unsafe proposals approved | Unsafe proposals presented to reviewers in controlled tests |
| Reject / modify / escalate rates | Decisions of each type | All reviewer decisions |
| Stale-approval invalidation coverage | Stale or replayed approvals rejected at execution | Stale or replayed approvals tested |
| Re-approval coverage after material change | Materially changed actions sent back to policy and review | Materially changed actions tested |
| Approval latency distribution | Time from request to decision, by percentile | All decided requests |
| False-deny rate | Expected-allow actions rejected or blocked | Expected-allow actions submitted |
| Abandonment / timeout rate | Requests with no decision within the window | All approval requests |
| Reviewer-context completeness | Requests showing every required field | All approval requests sampled |
| Audit-lineage completeness | Decisions with a complete linked record | All decisions sampled |
| Benign task utility | Benign tasks completed successfully | Benign tasks attempted, measured separately |
To practice these properties in a controlled setting, the Human Approval lab is a deterministic simulation of exact-action binding, reviewer evidence, stale approvals after material change, model-authored approval claims and fixed batch review.
Implementation Checklist
Before relying on human approval
- Authorization runs before the request and again at execution
- Approval requirement resolved by policy outside the model
- Approval bound to the fully resolved action and material state
- Material change invalidates approval
- Authoritative metadata and variable content rendered separately
- Reviewer eligibility and separation of duties defined per risk class
- Approvals single-use, task-bound and short-lived
- Timeouts fail safe for consequential actions
- Break-glass separately governed and audited
- Complete audit lineage per decision
- 01Generic confirm
"Allow this tool?" with no parameters shown.
- 02Model decides tier
The agent labels its own action as low risk.
- 03Approval override
Approval executes past a policy deny.
- 04Replay
A retry reuses an earlier approval.
- 05Chain approval
One approval covers every later step.
Standards and Sources
Standards and guidance referenced
- NIST
- AI RMF 1.0 GOVERN 3.2: roles and responsibilities for human-AI configurations and oversight, AI RMF 1.0 MAP 3.5: processes for human oversight are defined, assessed and documented
- OWASP
- LLM03:2026 Excessive Agency, Top 10 for Agentic Applications 2026 (neighbouring risk framework)
- Microsoft
- Agent Framework: Agent Safety and tool approval (vendor framework guidance), AI agent shared responsibility model (vendor guidance), Cloud Security Benchmark v2 AI-5 Ensure human-in-the-loop (vendor benchmark)
On source use: NIST AI RMF is cited for the governance principle that human oversight roles and processes are defined and documented; it does not define a specific HITL control, and none is implied. OWASP entries are risk frameworks, not HITL standards. Microsoft pages are vendor guidance: the Agent Framework safety page recommends approval for high-risk tools, and MCSB v2 AI-5 recommends human review and approval for critical actions; product-specific mechanisms are not generalized here.
Frequently Asked Questions
Is a confirmation dialog enough?
Only if it shows the exact bound action and authoritative metadata, and only for actions policy selects. A generic "allow?" prompt is a ritual, not a control.
Can the model decide when to ask for approval?
It may add a review, but it must never remove one. The requirement comes from deterministic policy outside the model.
Can a human approval override a policy deny?
No. Exceptions go through a separately governed break-glass path with its own eligibility and audit.
Is batch approval safe?
Only for an enumerated, fixed set of actions with a meaningful preview. Batching unrelated consequential actions invites rubber-stamping.
What happens if nobody responds?
For consequential actions the request times out and the action does not execute.
Does approving a plan cover the later steps?
No. Each materially distinct action or handoff is evaluated on its own, and each covered by the risk tier needs its own decision.
Who should be allowed to approve?
Authenticated reviewers whose eligibility for the risk class is resolved by policy, with separation of duties where the class warrants it.
Is the model's rationale evidence?
No. It is variable content, useful context at best, and it can be shaped by the same injection that shaped the action.
How does this relate to agent authorization?
Agent authorization decides whether an action may execute at all and remains authoritative. HITL adds a human judgement for selected permitted actions.
How do we avoid HITL becoming a bottleneck?
Reduce what needs review through least privilege, authorization and information flow control, tier by consequence, and measure latency and false denies alongside safety.
References
- AI RMF Core: GOVERN 3.2 and MAP 3.5NIST AIRC · StandardReviewed 2026-09-25
- LLM03:2026 Excessive Agency, OWASP Top 10 for LLM Applications 2026OWASP GenAI Security Project · GuidanceReviewed 2026-09-25
- OWASP Top 10 for Agentic Applications 2026OWASP GenAI Security Project · GuidanceReviewed 2026-09-25
- Agent Safety, Microsoft Agent FrameworkMicrosoft Learn · Technical documentationReviewed 2026-09-25
- Using function tools with human in the loop approvalsMicrosoft Learn · Technical documentationReviewed 2026-09-25
- AI agent shared responsibility modelMicrosoft Learn · GuidanceReviewed 2026-09-25
- Microsoft cloud security benchmark v2, Artificial Intelligence Security (AI-5)Microsoft Learn · GuidanceReviewed 2026-09-25