FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader#2175
FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader#21753nesdeniz wants to merge 10 commits into
Conversation
…-injection-boundary-dataset # Conflicts: # doc/bibliography.md
…-injection-boundary-dataset
|
This branch is now synchronized with current Fresh validation on the merged head:
The five GitHub Actions runs are currently |
adrian-gavrila
left a comment
There was a problem hiding this comment.
Looks great! Thanks for creating the PR, a couple of small comments and then we should be good to merge.
There was a problem hiding this comment.
Pull request overview
Adds a new remote SeedDatasetProvider loader for the CC BY 4.0 “Agentic Prompt-Injection Boundary Pairs” Hugging Face dataset, integrating it into PyRIT’s dataset discovery and documentation so users can reproducibly load paired prompt-injection boundary cases (attack-only by default, with typed filtering and pair reconstruction support).
Changes:
- Introduces
_AgenticPromptInjectionBoundaryDatasetwith typed enums for label/split/family/source-context filtering, pinned to an immutable HF revision, and with schema + pair-integrity validation. - Registers the loader and enums for discovery via
pyrit.datasets.seed_datasets.remote. - Adds focused unit tests plus documentation/bibliography updates to surface the dataset in the loading guide and citation lists.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/datasets/test_agentic_prompt_injection_boundary_dataset.py | Adds unit tests covering split loading, label modes, filters, and validation/error cases via mocked HF fetches. |
| pyrit/datasets/seed_datasets/remote/agentic_prompt_injection_boundary_dataset.py | Implements the new remote dataset loader, enums, validation, filtering, and SeedPrompt mapping with pinned HF revision metadata. |
| pyrit/datasets/seed_datasets/remote/init.py | Exports the new loader + enums so they’re discoverable/importable from the remote datasets package. |
| doc/references.bib | Adds the BibTeX citation entry for the dataset. |
| doc/code/datasets/1_loading_datasets.py | Documents the new loader in the dataset loading guide (jupytext source). |
| doc/code/datasets/1_loading_datasets.ipynb | Syncs the executed notebook with the new loader section and updated dataset-name listing output. |
| doc/bibliography.md | Adds the new citation key to the hidden citation-key dropdown list. |
Description
Closes #2174.
This PR adds a remote loader for the public CC BY 4.0 Agentic Prompt-Injection Boundary Pairs dataset:
The loader maps each row to a literal
SeedPromptand pins the immutable v1.0.0 Hugging Face revision. Before filtering, it validates required fields, label-dependent values, allowed families and source contexts, split metadata, duplicate IDs, and complete pair integrity.Family filtering uses
pair_familyso full-pair mode preserves both sides of a controlled pair. Agent-security decision labels and attack families remain in metadata rather than being misclassified as PyRIT harm categories. Pair members remain independent evaluation cases;pair_idpreserves the relationship without collapsing two prompts into oneSeedGroup.Source and release pipeline: https://github.com/3nesdeniz/agentic-prompt-injection-boundary-pairs
I am the dataset author and maintain the tagged source release. The dataset is synthetic, manually reviewed, and contains no customer or private data.
Validation
Revalidated after merging the current upstream
maininto this branch:git diff --checkpassed.The loader is registered in dataset discovery and documented in the loading guide, paired notebook, and project bibliography.