Security Objective
Output validation checks model or agent output before a downstream component uses it. A sink is anything that consumes that output: a tool call, an API request, a database operation, a rendered page, a file, a message or another agent. Validation compares the output with the sink's declared contract, such as schema, types, required fields, allowed values and formats, and either accepts or rejects it.
The objective is narrow. Validation prevents malformed, out-of-contract or unexpected output from reaching a sink that would misinterpret it. It does not decide whether an action is permitted, whether data may flow, or whether the output is true or wise.
Ownership Boundaries
| Control | Question it answers | Relation to output validation |
|---|---|---|
| Output validation | Is this output well-formed and within the sink's structural and format constraints | Owns parsing, schema and type checks, allowed values and formats, and accept or reject |
| Sink-specific encoding and parameterization | How is an accepted value handed to the sink so it cannot change the sink's grammar | Separate safe sink handling applied after validation |
| Agent authorization | May this fully resolved action execute for this principal | Deterministic and independent; valid output grants nothing |
| Information flow control | May data with these attributes reach this destination | Deterministic and independent; a well-formed destination is not a permitted one |
| Least privilege | How much authority exists at all | Limits the blast radius when valid but unwanted output gets through |
| Human-in-the-loop | Does a selected action need a human decision | Reviewers see the validated, resolved action, not raw model text |
| Prompt injection detection | Do correlated signals indicate an attack | Validation failures can be a detection signal |
| Input / content inspection | Does incoming content raise risk signals | Input side; validation is on the output side |
This matches the boundary used in agent authorization: validation covers structure, type and parameter shape; authorization covers permission.
Valid, Authorized and Safe Are Different Properties
A well-formed transfer request with a correctly formatted account identifier may still be forbidden.
A string field can be the right type and length and still carry text that the sink would interpret as instructions or code.
Checking that a value is a well-formed address, URL or path says nothing about whether that destination is allowed. Permission stays with authorization and IFC.
Validation accepts or rejects. Encoding, escaping and parameterization are separate steps that hand an accepted value to the sink safely.
Threat Model
Assume that model output can be influenced by injected content, tool results or other agents, and that the attacker's goal is to get a downstream sink to do something unintended: call a tool with altered parameters, send data to a new destination, render active content, or pass text to an interpreter. Also assume ordinary model errors produce malformed or out-of-contract output without any attacker.
Validation reduces the set of outputs that can reach a sink. It cannot recognize every harmful but well-formed value, which is why authorization, flow control and least privilege remain independent.
Trust Model
Model output remains variable throughout the flow. Validation establishes conformance to a declared contract, while authorization establishes permission; neither turns model-authored content into trusted data. Parsers and validators are system components; the values they produce are still variable data that has passed a shape check.
The same holds for output from critic, judge or detector models: their output remains variable even after validation against a fixed schema and consumption by a system decision.
Sink Inventory
| Sink | Validation focus | Separate safe handling |
|---|---|---|
| Tool or function call | Tool name from a fixed set, required typed arguments, no extra fields | Authorization of the resolved call |
| API request | Endpoint from an allowlisted operation set, typed body, value ranges | Authorization and IFC for destination and data |
| Database operation | Operation chosen from predefined operations, typed parameters | Parameterized queries; no model-written query text |
| Rendered page or markup | Expected content type and field structure | Context-appropriate output encoding; link and image policy |
| File write | File type, size, target within a declared area | Least privilege on the storage path |
| Message or email | Recipient field format, subject and body structure | Authorization and IFC on recipient and content |
| Another agent | Message schema for the handoff | The receiving agent treats it as variable input |
Structured Output and Declared Contracts
Validation is only as strong as the contract it checks. Declare a schema for each sink: allowed fields, types, required fields, value ranges, enumerations and maximum sizes. Structured output features in model APIs help models produce the expected shape, but the application must still parse and validate the result itself, because generation constraints are not a security boundary.
Parsing Strictness
Parse strictly. Reject output that does not parse, contains unknown or duplicate fields, has missing required fields or mixes structured content with surrounding prose. Do not attempt to extract a structure from free text by searching for something that looks like it, because an injected fragment can also look like it.
Allowlists Over Blocklists
Where the sink permits it, validate against allowlists: a fixed set of tool names, operation names, file types, units, currencies or formats. Blocklists of suspicious strings are incomplete by construction and are better treated as detection signals than as validation.
Executable Free Text Cannot Be Validated Into Safety
Validation does not make arbitrary model-generated SQL, shell commands, code, templates or other executable free text safe. Checking such text for length, characters or keywords leaves the interpreter's full grammar available to whatever the model produced.
Validation Versus Encoding, Escaping and Parameterization
| Step | What it does | Outcome |
|---|---|---|
| Validation | Checks the value against the sink contract | Accept or reject |
| Encoding / escaping | Represents an accepted value so the sink treats it as data in its specific context | Same value, safe representation |
| Parameterization | Passes an accepted value through a separate data channel of the sink API | Value cannot change the operation's structure |
Both are needed. A validated string rendered into markup still needs context-appropriate encoding; a validated identifier used in a query still belongs in a parameter, not in concatenated query text. Encoding and parameterization are established web security practice and are described conceptually here; this page gives no escape sequences or bypass examples.
Parameter Validation
Validate each parameter against its type, format, range and relation to other parameters: amounts within bounds, dates in the expected window, identifiers in the expected format, counts within limits. Resolve references, such as a record name to its identifier, through the application rather than accepting identifiers the model invented.
Destination Syntax Versus Destination Permission
Destinations need two separate checks. Validation confirms that an address, URL, path, channel or recipient identifier is well-formed, of the expected type and within the declared shape for the sink. Whether that destination is permitted for this principal, task and data is decided by authorization and information flow control.
Free-Text Fields Inside Valid Structures
Summaries, message bodies, comments and descriptions are legitimately free text. Validation can bound their size and encoding and confirm they sit in the right field. It cannot confirm their meaning. Treat them as variable data at every downstream consumer, including other models that read them, and apply encoding at render time.
Rendered Output and Implicit Requests
When output is rendered in a browser or client, some constructs can trigger requests without a user action, for example automatically loaded images or previews. Such requests can carry data to an external destination. Validation can restrict which output structures are allowed; the decision about which external destinations may be contacted belongs to flow and egress policy. Content that could reveal hidden context, as discussed in system prompt extraction, is a separate disclosure concern.
Multi-Step and Agent-to-Agent Output
In agent workflows, one step's output is another step's input. Validate at every boundary where output becomes a tool call, a handoff or a stored artifact, not only at the final response. For multi-agent systems, a validated handoff message is still variable content for the receiving agent.
Failure Behavior
A validation failure must never fall back to using the raw output. Reject the step, return a structured error to the orchestrator, and decide by risk tier whether to retry, degrade to a restricted mode or stop. Record the failure with the rejected output reference and the rule that failed.
Repair and Retry Loops
Asking the model to correct invalid output is common. Keep it safe: retry against the same schema, cap the number of attempts, and never relax the contract to make output pass. Automatic repair by the application, such as trimming or coercing values, must be deterministic, documented and limited to cases where the repaired value is unambiguous.
Relationship to Other Controls
| Control | May use validation to | Must not use validation to |
|---|---|---|
| Agent authorization | Evaluate a fully resolved, typed action | Treat a valid action as permitted |
| Information flow control | Receive a well-formed destination and data reference | Treat a well-formed destination as allowed |
| Human-in-the-loop | Show reviewers the exact validated action | Skip approval because validation passed |
| Prompt injection detection | Correlate validation failures with other signals | Treat passing validation as absence of attack |
| Spotlighting / data marking | Nothing directly; marking acts on input | Assume marked input produces valid output |
Worked Scenario
Synthetic example. A finance assistant can look up invoices and create payment drafts through a typed tool with fields for invoice identifier, amount, currency and payee account. It processes a vendor email whose body contains a synthetic test marker phrased as an instruction to pay a different account and to include a free-text query for "all unpaid invoices" in a note field.
| Gate | Observation | Outcome |
|---|---|---|
| Parser | Model output parses as the payment draft schema | Candidate accepted for validation |
| Validator | Amount within bounds, currency from allowlist, payee account well-formed, note within size limit | Valid; no decision about permission |
| Reference resolution | Invoice identifier resolved by the application; payee differs from the vendor master record | Mismatch recorded as a fact for downstream controls |
| Authorization | Payee is not an approved account for this vendor | Denied |
| Encoding | Note rendered in the draft view with output encoding; never executed | Displayed as text |
| Detection | Denial after external email entered context | Signal recorded |
In a benign variant, the email asks for payment of the listed invoice to the account on record. The output validates, the payee matches the master record, authorization allows a draft, and the draft goes to human approval as usual.
What the Scenario Proves and Does Not Prove
It shows validation passing a well-formed but harmful request, and authorization stopping it, which is the intended division of labour. It shows encoding keeping free text inert at render time. It does not show that validation detects manipulation, that the free-text note was harmless, or that the benign variant was safe for any reason other than the combination of controls.
Telemetry and Evidence
Record for each validation
- Correlation ID linking request, model output and sink call
- Sink, schema identifier and schema version
- Result (accepted or rejected) and the failing rule when rejected
- Reference to the raw output, stored with the same protection as the data it may contain
- Repair or retry attempts and their outcomes
- Downstream authorization and flow decisions recorded separately
Incident Response Relevance
When output reached a sink unexpectedly
- Preserve the model output, the validated value and the sink request
- Establish whether the value was valid, authorized and correctly encoded, as three separate questions
- Identify the schema gap or free-text path that allowed it
- Narrow the contract, add allowlists or replace free text with structured operations
- Add a sanitized synthetic case to the regression set
Controlled Testing
Test each sink contract with synthetic outputs: malformed structures, missing and extra fields, out-of-range values, wrong types, well-formed but unpermitted destinations and oversized free text. Run tests in isolated environments with harmless sinks and test identities. Verify both that validation rejects what it should and that authorization independently denies well-formed but unpermitted actions. Do not maintain a reusable library of injection strings for interpreters.
Metrics
Each measure keeps its own denominator. There is no composite score, and benign task utility is reported separately.
| Measure | Numerator | Denominator |
|---|---|---|
| Contract coverage | Sinks with a declared, versioned schema | Sinks that consume model output |
| Free-text sink exposure | Sinks that accept executable free text from models | Sinks that consume model output |
| Rejection correctness | Labelled invalid test outputs rejected | Labelled invalid test outputs |
| False rejection rate | Labelled valid test outputs rejected | Labelled valid test outputs |
| Independent denial rate | Well-formed but unpermitted test actions denied by authorization or IFC | Well-formed but unpermitted test actions |
| Raw fallback count | Executions that used unvalidated output | Executions that consume model output |
| Retry exhaustion rate | Steps that failed after maximum retries | Steps with at least one retry |
| Benign task utility | Benign tasks completed acceptably | Benign tasks attempted |
Common Mistakes
- 01Valid as permitted
Executing an action because it passed schema validation.
- 02Validating executable text
Keyword-checking model-written queries or commands and then running them.
- 03Validation instead of encoding
Assuming a validated string needs no encoding at render time.
- 04Destination policy in the validator
Deciding allowed domains or recipients inside format checks.
- 05Lenient parsing
Extracting structures from surrounding prose.
- 06Raw fallback
Using raw output when validation fails.
Implementation Checklist
Before relying on output validation
- Every sink that consumes model output has a declared, versioned schema
- Strict parsing with rejection of unknown, duplicate and missing fields
- Allowlists for operations, tools, formats and enumerations
- No executable free text; structured operations and parameterized calls instead
- Encoding and parameterization applied separately from validation
- Destination format validated here; destination permission decided by authorization and IFC
- No raw fallback; bounded retries against the same schema
- Validation at every step boundary in agent workflows
- Authorization, flow control and least privilege enforced independently
Standards and Sources
Standards and guidance referenced
- OWASP GenAI Security Project
- LLM01:2026 Prompt Injection
- OWASP Cheat Sheet Series
- Injection Prevention, Query Parameterization, Cross Site Scripting Prevention
- Microsoft
- Defend against indirect prompt injection attacks (vendor guidance)
On source use: OWASP LLM01:2026 is the current classification for prompt injection, the mechanism that can shape model output. OWASP covers the downstream risk of improperly handled model output as LLM10:2026 Improper Output Handling. The OWASP Cheat Sheets are established general web security guidance for parameterized operations, allowlist validation and context-appropriate output encoding; they are not specific to language models. Microsoft's guidance is vendor guidance recommending a defense-in-depth strategy that layers probabilistic and deterministic mitigations. No efficacy figures are claimed on this page.
Frequently Asked Questions
Does valid output mean the action is allowed?
No. Validation checks shape. Authorization decides permission independently.
Can validation make model-written SQL or shell commands safe?
No. Prefer structured operations and parameterized calls so the model supplies values, not executable text.
Is output encoding part of validation?
No. Validation accepts or rejects; encoding, escaping and parameterization are separate safe sink handling.
Should the validator decide which destinations are allowed?
No. It checks destination syntax and type. Permission stays with authorization and information flow control.
Are structured output features in model APIs enough?
They help produce the right shape, but the application must still parse and validate strictly.
What happens when validation fails?
The step is rejected, never executed with raw output. Retries use the same schema and are capped.
Do free-text fields need anything beyond validation?
Yes. They remain variable data for every consumer and need encoding where rendered.
Does validation detect prompt injection?
Not reliably. Failures can be a detection signal, but passing validation says nothing about the absence of an attack.
Where should validation run in agent workflows?
At every boundary where output becomes a tool call, handoff or stored artifact.
What limits damage when a valid but harmful action slips through?
Least privilege, authorization, flow control and, for selected actions, human approval.
References
- OWASP GenAI LLM Top 10 2026 (LLM01:2026 Prompt Injection)OWASP GenAI Security Project · GuidanceReviewed 2026-09-26
- Injection Prevention Cheat SheetOWASP Cheat Sheet Series · GuidanceReviewed 2026-09-26
- Query Parameterization Cheat SheetOWASP Cheat Sheet Series · GuidanceReviewed 2026-09-26
- Cross Site Scripting Prevention Cheat SheetOWASP Cheat Sheet Series · GuidanceReviewed 2026-09-26
- Defend against indirect prompt injection attacksMicrosoft Learn · GuidanceReviewed 2026-09-26