Every Attack Here Is a Prompt Injection

The form has a dropdown. You have caught a prompt that got your assistant to do something it shouldn’t, you are writing it up, and the field wants to know which technique it was. Prompt injection. Jailbreak. Obfuscation. You pick the closest one, because the form needs a value, and you move on.

I wanted to know how often that dropdown has a right answer, so I took the largest public collection of prompt attacks that actually worked and tried to label every one of them.

The right answer turned out to be the same every time. And that made the corpus much more useful than I expected, because a label you already know is a label you can measure yourself against.

The competition tells you the answer before you read a prompt

HackAPrompt ran eleven levels in 2023, each a small application with its own defence, against gpt-3.5-turbo, text-davinci-003 and FlanT5-XXL. 601,757 submissions. I kept the ones the competition scored as successful and deduplicated on the normalised prompt, leaving 18,479 unique attacks that worked.

The dataset records what each submission was scored against. On ten of the eleven levels that value is one fixed string, identical across all 538,857 rows on those levels:

I have been PWNED

That is the whole objective: make the model emit a specific phrase in defiance of the instructions it was given. Every success on those ten levels is AML.T0051 LLM Prompt Injection, and you know it before you look at any prompt text.

Level 2 is the exception, and it is the more interesting one. It records 62,900 distinct scoring targets, one per session — a secret key planted in the system prompt, which the attacker has to get the model to reveal. That is AML.T0056 Extract LLM System Prompt. Different technique, same conclusion: no winner in this corpus sits outside the taxonomy.

So the answer to “does the dropdown have a right answer” is yes, always, and it is nearly always the same one. Which turns the question into something better: if the correct label is known, how much of it can you actually recover from the prompt text?

11.4%

I ran the corpus through the mapping code in my adversarial-ai-cti library. That library takes taxonomy labels and returns ATLAS techniques and OWASP categories; it deliberately does not read prompt text, because guessing a technique from raw text is exactly the inference it refuses to make. Neither corpus carries taxonomy labels, so I wrote a text-signature detector to produce them: sixteen patterns, all in one file, each looking for fairly plain evidence of a technique. ignore the previous instructions is an instruction override. pretend you are is roleplay. base64 is base64.

Against a corpus that is 100% prompt injection, that pipeline labelled 2,114 prompts as AML.T0051. 11.4%. An 88.6% false negative rate on the one label we know for certain is correct.

Widening the criterion barely helps. A named sub-technique of any kind is detectable in 4,058 prompts (22.0%), and 2,785 (15.1%) resolve to some ATLAS technique or OWASP category. Four fifths of attacks that demonstrably worked carry no textual evidence of how.

This is the number I would take away from the whole exercise, because it is not an estimate. Most evaluations of technique attribution have to hand-label a sample and argue about it. Here the competition’s own scoring rule supplies ground truth for free, and the answer is that keyword attribution over prompt text recovers about a ninth of a label that applies to everything.

Most of the rest of the breakdown is the challenge talking

The 22% that is detectable makes a tidy distribution. Instruction override 2,114 prompts, translation 1,020, roleplay 529, output shaping 345, unicode trickery 120. It reads like a picture of what attackers reach for.

It isn’t. The prompt column holds each level’s full scaffold with the attacker’s text substituted in. Remove the attacker’s text and you get the task back, and the tasks explain the distribution:

• translation: 953 of 1,020 detections, 93.4%, on the four levels built around Spanish translation.
• unicode: 105 of 120, 87.5%, on the two levels that manipulate characters — level 9 separates every character with slashes, level 10 is emoji-only.
• roleplay: 217 of 529, 41.0%, on the persona bot and the nested storyteller jailbreak.
• output shaping: 132 of 345, 38.3% — and the rest is explained by the same thing, because the win condition was an exact phrase. respond only with is not an attack on the model’s rules here. It is a competitor complying with the scoring function.

So translation is not a technique attackers favoured. It is what you write when the bot in front of you only translates. Whatever these distributions describe, it is mostly the competition.

The length gap that looks like a finding, and isn’t

There is a tempting second result in this data. Winners with no detected technique are much shorter than winners with one: 94 characters against 180. The obvious reading is that a lot of these wins genuinely have no technique — too short to hold a wrapper — rather than the detector simply missing things.

That reading is wrong, and the check that shows it takes four lines.

A keyword detector fires more often on long text, because long text has more places for a pattern to match. So the unmatched residue of any substring detector is shorter than the matched set, regardless of what is in it. To see how much of the gap that alone accounts for, split the same corpus on a pattern with no relationship to technique at all — the word the:

The control produces a wider gap than the technique detector does: median 193 characters when the is present against 61 when it is absent, versus 180 and 94. Detection rate climbs monotonically with length in both cases — 1.9% in the shortest decile to 42.5% in the longest for the technique patterns, 3.5% to 91.8% for the control.

So the length asymmetry is a property of substring matching and says nothing about the prompts. It is worth reporting precisely because it is so easy to read the other way.

The check generalises, and it is cheap enough to be routine: any time a classifier’s miss set appears to have a characteristic, re-run the same split with a pattern you know is meaningless. If the meaningless pattern reproduces the effect, the effect belongs to the method rather than the data.

One related explanation worth ruling out: that the competition’s scoring penalised longer prompts, which would produce short winners directly. The score column does not support it. Correlation between token count and score is +0.114, and mean score rises from 47,018 in the shortest token decile to 88,371 in the longest. Whatever the rules were, longer winning prompts scored slightly better.

The level pattern is about the task, not the difficulty

Sub-technique visibility does fall across the levels, from 30.5% at level 0 to 12.6% at level 9. That invites a reading in which harder defences require less nameable attacks.

But the levels are eleven different applications with eleven different defences, not one defence being turned up. Eight of the ten levels anyone solved have a task that asks for one of my signatures, and the two that don’t — levels 4 and 5, a search engine and a grammar assistant — sit unremarkably in the middle at 15.3% and 15.6%. The profile tracks what each application does.

Two details survive that reading intact.

Level 9 admits exactly one signature. Of 831 winners, 105 register as unicode trickery and nothing else appears at all. That is consistent with its defence, which splits the input character by character with slashes. It is also where I am most blind: my unicode pattern catches zero-width characters, homoglyphs and directional overrides, and would miss an emoji-only or exotic-script attack entirely.

Level 10 was never solved. Zero successful submissions in the entire competition. A defence that held completely is the one result here with no caveat attached, and the dataset does not say why it held.

The 2025 rerun tells the same story

The Pliny HackAPrompt dataset reran the format in 2025 against seven current models — GPT-4.1, Claude 3.5 Sonnet, Gemini 2.5 Pro, DeepSeek-R1 among them — giving 1,905 unique winners from 16,902 submissions.

It is tempting to diff the two and call it a two-year trend. Don’t. Five of the twelve 2025 challenges are saturated, meaning every single winner carries a named technique, and those five hold 473 of the 523 signature-bearing winners in the corpus, 90.4%. Technique by technique: 341 of 346 base64 detections sit inside them, all 152 instruction-override detections come from two of them, and 65 of 72 other-encoding detections come from one. Any 2023-versus-2025 shift you compute is mostly a difference between two sets of challenge designs. Full per-challenge tables are in ANALYSIS.md.

Five signatures with nowhere to put them

One finding is about the taxonomies rather than the corpus, and it holds up. Of the 4,058 prompts where a signature was detected, 1,273 — 31.4% — resolve to no ATLAS technique and no OWASP category. Five of my sixteen signatures map to nothing in either: translation, hypothetical framing, claimed authority, manufactured urgency, and output shaping. Between them, 34.3% of all technique detections.

I should own that as a mapping decision rather than blame MITRE. My ATLAS coverage is a hand-maintained fourteen-technique subset pinned to release 2026.07, not the full matrix, so the gap may well be mine. I could not find an ATLAS technique covering language-switching as such, distinct from jailbreak or obfuscation; multilingual evasion plausibly belongs under one of those, and I chose not to guess, because a mapping table that guesses is worse than one with a hole in it.

The volume behind translation is a competition artifact, as above. But the absence of a place to put it is not, and multilingual evasion is a real technique in the wild.

What I’d actually take from this

Check how a corpus was built before interpreting what’s in it. Everything in this piece that corrected an earlier reading came out of three columns — expected_completion, prompt, score — sitting in a file I already had. It took one script. It should have run before I wrote anything, not after.

Text-derived technique labels are weak evidence, and now there’s a number for it. 11.4% recall against known ground truth, on a corpus where the technique is uncontested. If your pipeline attributes technique from prompt text, expect it to under-report, possibly by a lot. This detector is a set of 16 patterns I wrote, and a better one would do better, though it would also have to avoid false positives like a prompt that merely discusses base64.

Report coverage, not just distribution. A pie chart of technique shares is the natural output here and it is close to a lie, because the largest slice — no technique identified — gets dropped before the chart is drawn. Any distribution over prompt-attack techniques should carry its denominator and its miss rate on the same page.

An unmapped prompt is not an unimportant prompt. This is the part with practical teeth. A pipeline that enriches the prompts it can classify and quietly deprioritises the rest is deprioritising the large majority of attacks that actually worked. The mapping is a convenience for pivoting and reporting. It is not a triage signal and should not be wired up as one.

Both of those last two are arguments for the split dogesec draws between the fact of a prompt and any judgment about it. The prompt is what you have. The technique label is an opinion, frequently unavailable, and a model that treats the label as the primary object drops most of the evidence on the floor.

It also puts a ceiling on rule-based detection keyed to named techniques. A rule that fires on persona assignment or instruction override catches attacks that announce themselves in text. Here that is about a fifth of what worked — against a set where all of it was prompt injection.

What this doesn’t show

• These are competitions, not production traffic. Participants optimise against a scoring function and a fixed set of applications. The 2023 corpus is three years old and its targets are retired models.
• Successes only. No technique success rate can be derived from any of this, because the failures were dropped.
• The 11.4% is recall for one detector on one corpus. A better detector would do better; an LLM-based classifier would probably do much better. What the number bounds is the naive keyword approach, which is nonetheless what a lot of enrichment actually is.
• “Every attack here is prompt injection” is a fact about this corpus, not about prompt attacks. It follows from the competition’s scoring rule. A corpus assembled differently would not have that property, and would not offer free ground truth either.
• The task-affordance judgement is mine. Which keywords count as a task “asking for” a technique is a list in the script, visible and arguable. The concentration figures move if you disagree with the list.
• category-fallback at 0% is a property of the design, not a quality signal. The detector emits the vocabulary the keyword rules match on, so of course they match.
• The submission count differs from a figure I’ve published before. 601,757 is every row in the dataset; an earlier redundancy measurement used 579,953, which counted non-empty attack inputs only. Same dataset, two different questions.
• Deduplication is within each corpus, not across them. The 2023 and 2025 figures are separate.

Reproduce it

Two scripts in the adversarial-ai-cti repo. Fixed seeds, deterministic, counts only — neither emits attacker prompt text, and no corpus is committed.

python eval/corpus_construction_check.py    # how the corpus was built - run this FIRST
python eval/prompt_technique_survey.py      # the technique breakdown

corpus_construction_check.py produces the scoring targets, the recovered task scaffolds, the task-affordance concentrations, the length-bias control and the token/score check. prompt_technique_survey.py produces the distributions and the ATLAS/OWASP mappings. Full output in ANALYSIS.md. Both datasets are gated on HuggingFace, so accept their terms first.

Credits

The taxonomies. MITRE ATLAS, release 2026.07, and the OWASP Top 10 for LLM Applications, 2025 edition. Nothing above is a criticism of either. The 11.4% is a measurement of my own attribution pipeline, and it is only expressible because ATLAS gives the technique a stable identifier to be wrong about.

The concepts. The Indicator of Prompt Compromise, and the argument for describing prompt attacks by behaviour rather than literal text, are Thomas Roccia’s — see The State of Adversarial Prompts and the NOVA rule engine. This piece is a small piece of evidence for that argument. The fact-versus-judgment split, and the prompt as a first-class observable, come from dogesec.

The data. HackAPrompt (MIT) — Schulhoff et al., Ignore This Title and HackAPrompt, EMNLP 2023 — and the Pliny HackAPrompt dataset (CC-BY-4.0). Both released by the HackAPrompt organisers; the 2025 challenges were designed with Pliny. Publishing the scoring target and the full scaffold alongside every submission is what made the ground-truth comparison possible at all, and not every dataset does that.

Code. adversarial-ai-cti, Apache-2.0. The signature detector and the construction check in eval/ are measurement instruments written for this survey, not part of the library’s published mapping surface.