Tensor Trust - Human-Generated Prompt Injection Dataset

Overview

Tensor Trust collected adversarial examples from players of an online game in which each player defends an account with a prompt and tries to break other players' defenses. The result is a dataset of human-generated attacks and defenses against instruction-following LLMs, which the authors also turn into a benchmark.

Research question

What do human-created prompt injection attacks look like at scale, and how robust are instruction-following models against the strategies players discover?

Method and artifact

Attacks and defenses were gathered through gameplay and released as data. The authors derive two benchmark tasks from the dataset: prompt extraction (getting a model to reveal its instructions) and prompt hijacking (getting a model to follow the attacker instead of the defender). The current arXiv abstract reports over 126,000 attacks and 46,000 prompt-based defenses; earlier versions of the paper may report different counts, so this page cites only that version. This page does not reproduce any attack strings from the dataset.

• Tensor Trust project paper page (tensortrust.ai) - official project page linking the paper, data and code

Key findings reported by the authors

Source statements: the points below summarize what the cited publication reports for the models and configurations it tested.

• The authors describe the dataset as, to their knowledge at the time, the largest collection of human-generated adversarial examples for instruction-following LLMs.
• Attacks in the dataset show easily interpretable structure.
• Benchmark results indicate that many of the tested models were vulnerable to strategies from the dataset.
• Some attack strategies generalized to deployed LLM-based applications despite different constraints.

Security interpretation

Site interpretation: this section is Prompt-Injection.net's explanation of why the work matters, not a claim made by the authors.

The dataset is useful precisely because it is human-generated: it captures creative strategies that synthetic templates can miss. For defenders it illustrates why instruction secrecy and prompt-based defenses are weak security boundaries, which matches this site's guidance that system prompts should not hold secrets or sole authority.

Limits and caveats

• The game setting is narrow (short defenses, a single secret access condition) and differs from production applications.
• Model results are historical results for the models tested at the time.
• Dataset counts differ between paper versions; cite the version you use.
• The work focuses on direct injection; it does not measure indirect injection through retrieved content.

How to use this work

Use it as a source of realistic human attack diversity when designing evaluations or training detectors, and as evidence that prompt-based defenses alone are fragile. Handle the released data as research material within an authorized testing context.

Related content on Prompt-Injection.net

• Direct Prompt Injection - How attacker-controlled input competes with application instructions.
• System Prompt Extraction - How hidden application context is disclosed or reconstructed through model output, why a system prompt must never be a secret or a control boundary, and which deterministic controls decide the actual impact.
• Prompt Injection vs Jailbreaking - How the terms overlap, why usage varies across sources, and how to classify and report a case by source, delivery, challenged boundary and effect.
• System Prompts - What system prompts can and cannot do, why hidden is not secret, how to compose dynamic prompts safely, and which controls belong outside the model.
• Direct Prompt Injection Basics - A guided conceptual simulation of model context, competing instructions, and application authorization.
• Prompt Injection Detection - How to detect prompt injection attempts and effects at runtime by correlating content, context, model behavior, plan, tool and enforcement signals. Probabilistic evidence for response and investigation, never authorization or a replacement for deterministic enforcement.