# Evaluating Prompt Injection Datasets

Published on

May 7, 2025

Reading Time:

5

Minutes

By

Kevin Stangl, Andrew Davis

### Table of Contents

- [Introduction](/content/research/evaluating-prompt-injection-datasets#introduction/index.html)
- [The Bad: Limitations of Existing Prompt Injection Datasets](/content/research/evaluating-prompt-injection-datasets#the-bad-limitations-of-existing-prompt-injection-datasets/index.html)
- [The Good: Effective Public Datasets](/content/research/evaluating-prompt-injection-datasets#the-good-effective-public-datasets/index.html)
- [The Intriguing: Empirical Refusal Rates](/content/research/evaluating-prompt-injection-datasets#the-intriguing-empirical-refusal-rates/index.html)
- [What to Make of These Numbers](/content/research/evaluating-prompt-injection-datasets#what-to-make-of-these-numbers/index.html)
- [Conclusion](/content/research/evaluating-prompt-injection-datasets#conclusion/index.html)

## **Introduction**

Prompt injections, jailbreaks, and malicious textual inputs to LLMs in general continue to pose real-world threats to generative AI systems. Informally, in this blog, we use the word “attacks” to refer to a mix of text inputs that are designed to overpower or re-direct the control and security mechanisms of an LLM-powered application to effectuate a malicious goal of an attacker.

Despite improvements in alignment methods and control architectures, Large Language Models (LLMs) remain vulnerable to text-based attacks. These textual attacks induce an LLM-enabled application to take actions that the developer of the LLM (e.g., OpenAI, Anthropic, or Google) clearly do not want, ranging from emitting toxic content to divulging sensitive customer data to taking dangerous action, like opening the pod bay doors.

In particular, in attacks known as _prompt injections_, a malicious user prompts the LLM so that it believes it has received new developer instructions it must follow. These untrusted instructions are concatenated with trusted instructions. This co-mingling of trusted and untrusted input can allow the user to twist the LLM to his or her own ends. Below is a representative prompt injection attempt.

**_Sample Prompt Injection_**

These attacks play on the instruction-following ability of LLMs to induce unauthorized action. These actions may be dangerous and inappropriate in any context, or they may be typically benign actions _which are only harmful in an application-specific context. This dichotomy is a key aspect of why mitigating prompt injections is a wicked problem._

## **The Bad: Limitations of Existing Prompt Injection Datasets**

How should one evaluate a prompt injection defensive solution? A typical approach is to download benchmark datasets from public sources such as HuggingFace and assess detection rates. We would expect a high True Positive Rate (recall) for malicious data and a low False Positive Rate for benign data.

### Drawbacks

- **Staleness:** Datasets quickly become outdated as defenders train models against known attacks, resulting in artificially inflated true positive rates.
- **Labeling Biases:** Dataset creators often mix distinct problems. For example, prompts that request the LLM to generate content with clear political biases or toxic content.
- **CTF Over-Representation:** Capture-the-flags (CTFs) are contests where white-hat hackers attempt to break a system and test its defenses, but they do not align well with real-world data.

## **The Good: Effective Public Datasets**

To navigate the complex landscape of public prompt injection datasets, we offer data recommendations categorized by quality.

### **Recommended Datasets**

- [**qualifire/Qualifire-prompt-injection-benchmark**](https://huggingface.co/datasets/qualifire/Qualifire-prompt-injection-benchmark)
  - Size: 5,000 rows  
  - Language: Mostly English  
  - Labels: 60% benign, 40% jailbreak

- [**xxz224/prompt-injection-attack-dataset**](https://huggingface.co/datasets/xxz224/prompt-injection-attack-dataset)
  - Size: 3,750 rows  
  - Language: Mostly English  
  - Labels: None

- [**yanismiraoui/prompt_injections**](https://huggingface.co/datasets/yanismiraoui/prompt_injections)
  - Size: 1,000 rows  
  - Languages: Multilingual  
  - Labels: None

- [**jayavibhav/prompt-injection-safety**](https://huggingface.co/datasets/jayavibhav/prompt-injection-safety)
  - Size: 50,000 train, 10,000 test rows  
  - Labels: Benign (0), Injection (1), Harmful Requests (2)

## **The Intriguing: Empirical Refusal Rates**

A refusal is when an LLM thinks a request is malicious based on its post-training and declines to answer or comply with the request.

| **Dataset**  | **Label** | **GPT-4o** | **Claude 3.7 Sonnet** | **Gemini 2.0 Flash** | **Average** |
|--------------|-----------|-------------|------------------------|----------------------|-------------|
| Casual Conversation | 0 | 1.6% | 0% | 4.4% | 2.0% |
| Qualifire | 0 | 10.4% | 6.4% | 10.8% | 9.2% |
| Hackaprompt | 1 | 30.4% | 24.0% | 26.8% | 27.1% |
| Yanismiraoui | 1 | 72.0% | 32.0% | 74.0% | 59.3% |
| Qualifire | 1 | 73.2% | 61.6% | 63.2% | 66.0% |

## **What to Make of These Numbers**

Evaluating model security is complex, especially as attackers continuously adapt. Public datasets remain essential, but their limitations must be clearly understood. Recognizing these shortcomings and leveraging the most reliable resources available enables more accurate assessments of generative AI security.

## **Conclusion**

Evaluating prompt injection defensive models is complex, especially as attackers continuously adapt. Public datasets remain essential, but their limitations must be clearly understood. Recognizing these shortcomings and leveraging the most reliable resources available enables more accurate assessments of generative AI security.  
HiddenLayer is responding to this security challenge today.