Overview
This paper addresses prompt injection from the model side. It argues that a core weakness is that models often treat developer instructions with the same priority as text from users or third parties, and proposes training models to respect an explicit hierarchy.
| Field | Value |
|---|---|
| Resource type | Paper |
| Original title | The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions |
| Authors | Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, Alex Beutel |
| First public version | 2024-04-19 |
| Publication status | OpenAI research publication, also available on arXiv (arXiv:2404.13208). |
| Focus areas | Model training, Defenses, Direct injection, Indirect injection |
| Artifacts | No code or data artifact listed |
| Last reviewed | 2026-09-26 |
Research question
Can a model be trained to follow higher-privilege instructions (for example from the system message) and selectively ignore conflicting lower-privilege instructions (for example in tool output), without losing general capability?
Method and artifact
The authors define how a model should behave when instructions of different privilege conflict, and propose a synthetic data generation method that demonstrates this behavior for training. They apply the method to GPT-3.5 and evaluate robustness against several attack types, including some not seen during training, as well as standard capability benchmarks. No public code or dataset artifact is listed here.
Key findings reported by the authors
Source statements: the points below summarize what the cited publication reports for the models and configurations it tested.
- The authors identify equal treatment of instructions from different sources as a primary vulnerability underlying prompt injections and jailbreaks.
- Training with the proposed data reportedly increases robustness drastically for the tested model, including against attack types not seen during training.
- They report minimal degradation on standard capabilities.
Security interpretation
Site interpretation: this section is Prompt-Injection.net's explanation of why the work matters, not a claim made by the authors.
Instruction hierarchy is a model-facing robustness measure: it can make injection less likely to succeed, but the result is still a learned tendency, not an enforced rule. That is why this site pairs it with application-side controls such as authorization, least privilege and information flow restrictions, which hold even when the model is persuaded.
Limits and caveats
- Published experiments apply the method to GPT-3.5; results do not automatically carry over to other models or later versions.
- Robustness is measured on specific evaluations; adaptive attackers may find gaps.
- The method does not give applications a verifiable guarantee about which instructions were followed.
How to use this work
Use it to understand why message roles matter and why putting untrusted content in the highest-privilege position is harmful. When designing systems, treat model-side hierarchy as one layer and keep authority decisions in application code.
Related content on Prompt-Injection.net
- Instructions and Instruction Hierarchies - How models prioritize competing instructions, why provider schemes differ, and why precedence is behavior rather than authorization or enforcement.
- System Prompts - What system prompts can and cannot do, why hidden is not secret, how to compose dynamic prompts safely, and which controls belong outside the model.
- Direct Prompt Injection - How attacker-controlled input competes with application instructions.
- Indirect Prompt Injection - How external content can carry instructions into an AI system.
- Agent Authorization - How to authorize AI agent actions outside the model: initiating principal, agent identity, resource, action, parameters, destination, scope and policy version, with exact-action binding and downstream revalidation.
- Least Privilege - How to reduce the size, reach and lifetime of agent authority before any action is evaluated: identities, scopes, tool exposure, data access, JIT elevation, review and revocation readiness.