Adversarial Testing

Security Objective

Adversarial testing for prompt injection is an authorized assurance activity in which a team exercises its own AI system, controls, telemetry and recovery paths with representative adversarial conditions. Its objective is to learn where trust boundaries hold, where they fail, and whether failures would be observed and recoverable, before those conditions arise outside the test.

Threat Model

Testing assumes the same threat model as the controls it exercises: untrusted language can reach model context through direct input, external content, retrieval, tool output, delegated agents and persistent state; model behavior is variable; probabilistic detectors miss some events and flag some benign ones; and deterministic controls outside the model are expected to limit what influenced behavior can do.

Testing adds its own risks. A test can cause real side effects, leak real data into test logs, or grow into an unmanaged payload repository. The program design below treats these as risks to control, not as acceptable costs.

Adversarial Testing, Labs and Incident Response

The Test Case Library provides reviewed, safe test cases for each testing domain, expanded selectively where a distinct boundary needs its own case.

The Labs illustrate individual ideas and are not a test suite for any product. The Prompt Injection Security Testing methodology provides the practitioner foundation for evidence states, local pass criteria and reporting across real systems. Incident Response handles real events; its tabletop exercises are one input to testing, and findings from incidents feed regression tests here.

What Testing Can and Cannot Prove

Scope and Authorization

Testing is authorized, scoped and governed. A written scope names the owner, systems, environments, identities, tools, destinations, data classes, time window, stop conditions and escalation contacts. Testing grants no permission beyond that scope: a tester's role does not authorize actions the scope excludes, and a model or agent can never widen it.

Test Environment Choices

Many organizations should never test adversarially in production. If they do, synthetic identities, synthetic destinations and reversible actions remain mandatory.

Test Asset Inventory

Trust-Boundary Coverage Model

Coverage is organized around trust boundaries rather than around clever inputs. Each boundary where untrusted content enters context, or where a model proposal becomes an effect, needs tests that exercise it in both expected-allow and expected-deny directions.

Test Case Anatomy

A test case is a versioned artifact. The following record is engineering guidance, not a standard schema. It does not require collecting model chain-of-thought.

Expected Outcome and Oracle Design

An oracle decides whether a run passed. Oracles for deterministic controls read enforcement records and downstream state: the authorization decision, the broker log, the synthetic sink. Oracles for model behavior classify outputs across repeated runs. A transcript alone is a weak oracle for a control, because a model can claim an action it never performed or omit one that happened.

Model-Behavior Assertions vs Security-Control Assertions

Do not merge these into one score. A high model-deviation rate with perfect control enforcement is a different risk position from a low deviation rate with a missing authorization check.

Direct Injection Tests

Direct tests exercise the path where the requesting user supplies instruction-like text. The useful questions are whether the application's instruction hierarchy and policy hold, and whether any deviation remains bounded by the requesting principal's own authority. See Direct Prompt Injection for the attack mechanics.

Indirect Content Tests

Indirect tests place synthetic instruction-like text in web, email, document or search content that the system processes on behalf of a different principal. Tests verify that provenance survives extraction and that downstream decisions still use the initiating principal's authority. See Indirect Prompt Injection.

RAG and Retrieval Tests

Retrieval tests check that relevance does not become trust: that tenant and access filters apply before retrieval, that source metadata reaches policy, and that a synthetic document from a lower-trust source cannot change what the application authorizes.

Tool-Output Tests

Tool-output tests return synthetic results containing instruction-like text and verify that the result is treated as data, that provenance is recorded, and that any subsequent proposal is authorized on its own merits.

Agent and Action Tests

Action tests exercise the step where a proposal becomes an effect. Each should record proposed, authorized, submitted, executed, succeeded, failed and unobserved as distinct states. A proposal is not execution, and an unobserved stage is not evidence that nothing happened.

Multi-Agent and Delegation Tests

Delegation tests verify that a child task receives only the capabilities the parent was allowed to delegate, that returned results carry provenance, and that parent and child identifiers correlate. A finding in a child does not by itself establish a finding in the parent, or the reverse.

Authorization and Least-Privilege Tests

Authorization tests submit proposals the principal may and may not perform and assert the authorization decision exactly. Least-privilege tests confirm that capabilities outside the task simply do not exist for the test identity, rather than relying on a denial.

Information-Flow and Tenant-Boundary Tests

Flow tests label synthetic data by source and classification and verify that information flow control blocks disallowed movement to destinations, tenants or models, including through summaries or derived artifacts.

Egress Tests

Egress tests use synthetic destinations inside and outside the allowlist and verify that egress restrictions deny before connection, recheck after resolution and redirects, and cannot be bypassed by a worker's direct network path.

Output-Validation and Sink-Handling Tests

Validation tests submit structurally invalid values and confirm output validation rejects them before the sink. Passing validation shows structure and format, not trust or semantic safety; sink-specific encoding is tested separately.

Sandboxing and Containment Tests

Containment tests are defensive regression classes: host paths are inaccessible, denied networking is unavailable, quotas terminate runaway work, read-only inputs stay unchanged, broad credentials are absent, export requires release, and unavailable sandboxing fails closed. They do not teach escape techniques.

Detection, Plan Drift and Tool-Chain Telemetry Tests

Telemetry tests verify that detection, plan drift and tool-chain signals are emitted, correlated and routed. Detector recall and false-positive rates are measured over repeated runs. A missed detection with successful enforcement is a detector gap, not a security failure of the enforcement layer.

Incident-Response and Evidence-Capture Tests

These tests check whether a synthetic run can be reconstructed end to end from provenance and decision records, whether evidence is preserved before cleanup, and whether containment actions such as revoking a test token work as documented.

Persistence, Memory and Cross-Run State Tests

Persistence tests write synthetic, model-influenced state into memory, caches or vector stores and verify that a later run treats it as untrusted, that lineage identifies which runs consumed it, and that reset restores known-good state without affecting unrelated tenants.

Repetition and Nondeterminism

A single run is not enough for probabilistic behavior. Repeat model-behavior cases enough times to report a rate with its run count, and record versions and seeds where supported. Deterministic controls should give the same decision for the same input and configuration; variance there is itself a finding.

Coverage and Traceability Matrix

A practical matrix maps entry vector, capability, boundary control, expected decision and observed state. This is engineering guidance, not a standard.

Every cell needs expected-allow cases as well as expected-deny cases. A system that blocks everything is not a successful security design.

Safe Synthetic Fixtures and Benign Markers

Use fictional documents, fake identities, synthetic destinations and benign marker strings whose appearance in an output or sink shows that influence reached that point. Never require real secrets, real customer data, real external targets, real money movement, destructive changes or uncontrolled autonomous execution.

Regression fixtures derived from incidents or research are sanitized to the minimum needed to exercise the boundary, so the suite does not become a payload repository.

Findings, Evidence States and Severity

Findings keep evidence states separate: model influence observed, control decision recorded, action submitted, action executed, effect confirmed on a synthetic sink, telemetry missing. A detector alert is evidence, not a finding of compromise.

Severity follows reachable impact, not prompt cleverness: which privileges, data classes, destinations, tenants and persistent state a failure could reach given the controls that actually held. An elaborate input that only changes wording is lower severity than a simple one that reaches an unauthorized write.

Regression Suite Lifecycle and Change-Triggered Retesting

Regression cases come from documented incidents, research findings, red-team findings and newly discovered trust-boundary failures. Each is sanitized, owned, versioned and retired when the boundary it tests no longer exists.

Worked Scenarios

All scenarios use fictional systems, synthetic fixtures and benign markers. No attack text is shown.

Metrics With Denominators

These are operational measures, not proof of security. Do not combine them into one aggregate security score, and avoid vanity counts such as the number of prompts tried.

Common Mistakes

Implementation Checklist

Standards and Sources

Source scope: NIST AI 100-2e2025 provides broad adversarial machine learning terminology and taxonomy; it is not a prompt-injection test standard. The OWASP GenAI Red Teaming Guide gives general guidance for red teaming generative AI systems; it does not define the coverage matrix, record schema or metrics on this page. OWASP LLM01:2026 and the Agentic Top 10 are risk context. MITRE ATLAS is used for threat-knowledge orientation, not as evidence that any test is effective. No cited source defines this adversarial testing model as a complete standard. The coverage matrix, test case record, scenarios, severity approach and metrics are this site's engineering guidance.

Frequently Asked Questions