Prompt-Injection.net
ENDE
Learn

Trust Boundaries and Model Context

What model-visible context is, why source, representation and trust are different things, why position in a prompt never creates trust, and why authorization, information flow control and output validation must sit outside the model.

By Norbert HofmannPublished: 2026-09-26Last reviewed: 2026-09-26Level: FundamentalsAudience: Developers, AI Engineers, Security Engineers
Area
Learn - architecture foundation
Core idea
Trust comes from system evidence, never from prompt position
Key boundary
Context assembly
Controls outside the model
Authorization, information flow control, output validation
Supports
Direct Prompt Injection Basics lab
Table of contents

What model-visible context is

Model-visible context is everything the model receives for one inference: instructions from the application, the conversation, retrieved passages, tool results, file contents, and any metadata the application chose to render as text. From the model's perspective it is one sequence of tokens. Whatever structure the application intended must either survive as text the model interprets, or be kept outside the model entirely.

Source, representation and trust

Three things are easy to conflate:

Three separate properties
PropertyMeaningWho establishes it
SourceWhere content originated: a user, a mailbox, a web page, a database row, a toolThe application, from system evidence
RepresentationThe form the content takes in context: extracted text, a summary, a tool result, a rendered tableThe parsing, retrieval or transformation step
TrustHow much authority the application grants the content for a given decisionApplication policy, based on the source and its evidence

A trusted source does not guarantee a trustworthy representation, and neither makes the content's instructions authoritative. A document from an internal repository may still contain text written by an outside party. See Data Provenance for how origin and transformations are recorded.

Trust does not arise from position or concatenation

Placing content inside a system message, near the top of the prompt, or inside a delimited block does not make it trusted, and placing it lower does not make it harmless. Concatenation merges strings; it does not carry permissions with them. Once text is in context, the only remaining question the model answers is how to continue, and that answer is influenced by all of it.

System, developer, user, retrieved and tool content

Applications typically combine content from several origins. Vendors expose different role models, names and priority rules, so the categories below describe origins, not a universal API:

Application instructionsTrusted

Written and deployed by the application owner. Trusted as configuration, but not secret and not a control boundary.

Authenticated user inputVariable trust

The user is authenticated; their text is still variable input that may be mistaken or adversarial.

Retrieved contentUntrusted

Search results, documents, web pages and knowledge-base passages written by others.

Tool outputUntrusted

Results returned by tools. A trusted tool can return content authored by anyone.

Trust labels here describe authority for instructions, not factual accuracy. Treat them as defaults the application overrides only with system evidence.

Application-controlled metadata vs model-visible prose

The application knows facts the model cannot verify: the authenticated user, the tenant, the document's access list, which tool produced a result, whether content passed a check. Keep these facts as structured metadata in application state and use them in decisions. If some of it is also shown to the model, the rendered copy is informational; the authoritative copy stays in the application.

Context assembly as a security boundary

Context assembly is the step where the application decides what enters the model's context: which documents, which tool results, how much history, which metadata. It is a real security boundary because it determines what can influence the model and what data the model can expose.

  • Include only what the task needs. Every extra document is extra attack surface and extra exposure.
  • Apply access control before retrieval, so the model never sees data the user could not read.
  • Record what was assembled and from where, so decisions and incidents can be reconstructed.

Provenance and lineage

Provenance records where content came from; lineage records how it was transformed on the way into context. Both belong to application metadata, travel alongside the payload as separate objects, and let later controls ask: does this proposal depend on content from an external source? Data Provenance covers the details.

Retrieved and tool output stays variable

Retrieved passages and tool results remain variable or untrusted content unless system evidence says otherwise. Relevance ranking is not trust. A tool's authentication proves who answered, not that the answer is safe to follow. See RAG Prompt Injection, Tool Output Injection and Indirect Prompt Injection.

Controls that sit outside the model

Because the model cannot enforce boundaries, the decisions that matter happen in deterministic code around it:

Where each decision lives
ControlDecidesUses
Agent AuthorizationIs this exact action permitted for this principal and resource?Identity, scope, policy
Information Flow ControlMay this data reach this destination or influence this action?Labels and provenance
Output ValidationIs this output well-formed for its sink?Schemas and sink constraints

Validation accepts or rejects structure; authorization and flow control decide permission. None of them turns model-authored content into trusted content. See Agent Authorization, Information Flow Control and Output Validation.

A conceptual trust flow

From sources to effect
Variable trustExternal or user content
SystemProvenance metadata
SystemContext assembly
SystemModel
Variable trustProposal
SystemAuthorization and IFC
Variable trustEffect

System components make decisions; content and proposals stay variable throughout. The effect is whatever the deterministic checks permitted, not whatever the model wanted.

Trust by position
  1. Content in prompt
  2. Model decides
  3. Action runs

The model's output is treated as the decision.

Trust by system evidence
  1. Content plus metadata
  2. Model proposes
  3. Policy decides
  4. Action runs if permitted

The model proposes; application policy decides.

Common mistakes

Patterns to avoid

  • Treating the system prompt as a secret or as the enforcement layer.
  • Passing identity or permissions to the model as text and trusting its echo.
  • Assuming content from a trusted tool or internal store is trusted content.
  • Retrieving first and filtering access afterwards.
  • Letting the model choose destinations without an independent check.
  • Dropping provenance when content is summarized or transformed.

For the system prompt case specifically, see System Prompt Extraction.

Implementation checklist

Before you ship

  • Every content item entering context has a recorded source.
  • Access control is applied before retrieval and tool execution.
  • Authoritative facts live in application state, not in model text.
  • Every consequential action passes a deterministic authorization check.
  • Destinations are checked by information flow policy.
  • Output is validated for its sink before use.
  • Assembled context and decisions are logged for reconstruction.

See it in the lab

The Trust Boundaries and Model Context lab turns this article into a hands-on exercise: compare model-visible prose with application evidence and apply policy to a model proposal. The Direct Prompt Injection Basics lab makes model-visible context and the authorization step visible in a controlled simulation. Start with What Is Prompt Injection? if you have not yet, and continue with the Defense overview.

Standards and sources

How this article uses sources

OWASP
GenAI LLM Top 10 2026 - LLM01:2026 Prompt Injection as a named risk
NIST
AI 100-2e2025 - terminology only
Microsoft
Defend against indirect prompt injection attacks - untrusted external content, isolation of untrusted content with metadata, defense in depth

Frequently asked questions

Is the system prompt trusted?
It is trusted configuration from the application owner, but it is neither secret nor an enforcement boundary.
Does putting content in a separate message role make it safe?
Roles help models prioritize, but they are behavioral signals. They do not enforce anything, and vendors implement them differently.
Is content from our own database trusted?
Only as far as system evidence supports it. Internal storage can hold text written by external parties.
Why keep metadata outside the prompt?
Because decisions must use facts the model cannot alter or misstate.
What is the single most important boundary?
There is no single one. Context assembly limits influence and exposure; authorization and flow control decide effects.
Does output validation make model output trusted?
No. It checks structure for a sink. Output remains variable data.

References

  1. OWASP GenAI LLM Top 10 2026OWASP GenAI Security Project · GuidanceReviewed 2026-09-26
  2. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2e2025NIST · Standard · 2025-03Reviewed 2026-09-26
  3. Defend against indirect prompt injection attacksMicrosoft Learn · GuidanceReviewed 2026-09-26