FEAT: Add PuzzledConverter (word-puzzle jailbreak from arXiv:2508.01306)#2256
Open
shashank03-dev wants to merge 1 commit into
Open
FEAT: Add PuzzledConverter (word-puzzle jailbreak from arXiv:2508.01306)#2256shashank03-dev wants to merge 1 commit into
shashank03-dev wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a new text-to-text converter, PuzzledConverter, implementing the PUZZLED word-puzzle jailbreak technique. It fits into PyRIT’s converter stack by masking selected sensitive words and re-encoding them as a solvable puzzle (word search / anagram / crossword), optionally augmenting deterministic clues with best-effort LLM-generated semantic hints.
Changes:
- Added
PuzzledConverterplus supporting keyword-masking and puzzle-building utilities underpyrit/converter/puzzled/. - Added a new converter seed prompt template (
pyrit/datasets/converters/puzzled_converter.yaml) and exported the converter frompyrit.converter. - Added unit tests and documentation updates, including new citations and a usage example.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/converter/test_puzzled_puzzle_builders.py | Unit tests for anagram/crossword/word-search puzzle builders. |
| tests/unit/converter/test_puzzled_keyword_masker.py | Unit tests for keyword selection, masking behavior, and spaCy fallback/caching. |
| tests/unit/converter/test_puzzled_converter.py | Unit tests for converter construction, determinism, puzzle fallback behavior, and optional LLM-clue path. |
| pyrit/datasets/converters/puzzled_converter.yaml | New prompt template used to scaffold the puzzle-wrapped instruction. |
| pyrit/converter/puzzled/puzzled_converter.py | Core PuzzledConverter implementation, including optional semantic clue generation + caching. |
| pyrit/converter/puzzled/puzzle_builders.py | Deterministic puzzle-building utilities and PuzzleType enum. |
| pyrit/converter/puzzled/keyword_masker.py | Keyword ranking/masking logic with spaCy POS tagging fallback. |
| pyrit/converter/puzzled/init.py | Subpackage exports for the PUZZLED components. |
| pyrit/converter/init.py | Exposes PuzzledConverter via the main converter package API. |
| doc/references.bib | Adds BibTeX entry for the PUZZLED paper. |
| doc/code/converters/1_text_to_text_converters.py | Adds a short PuzzledConverter usage example in the docs notebook source. |
| doc/code/converters/1_text_to_text_converters.ipynb | Mirrors the docs example in the paired notebook. |
| doc/bibliography.md | Adds the PUZZLED citation key to the hidden citations list. |
Comment on lines
+138
to
+149
| global _nlp, _nlp_loaded | ||
| if _nlp_loaded: | ||
| return _nlp | ||
| _nlp_loaded = True | ||
| try: | ||
| import spacy # type: ignore[ty:unresolved-import] | ||
|
|
||
| _nlp = spacy.load("en_core_web_sm") | ||
| except Exception: | ||
| logger.info("spaCy model 'en_core_web_sm' unavailable; using length-based keyword selection instead.") | ||
| _nlp = None | ||
| return _nlp |
Comment on lines
+237
to
+242
| cache_key = tuple(words) | ||
| if cache_key in self._clue_cache: | ||
| return self._clue_cache[cache_key] | ||
|
|
||
| instruction = _CLUE_GENERATION_INSTRUCTION.format(words=", ".join(words)) | ||
| request = Message( |
| } | ||
|
|
||
|
|
||
| class PuzzledConverter(Converter): |
Comment on lines
361
to
+365
| "# CodeChameleon [@lv2024codechameleon] encrypts and wraps in code\n", | ||
| "code_chameleon = CodeChameleonConverter(encrypt_type=\"reverse\")\n", | ||
| "print(\"CodeChameleon:\", await code_chameleon.convert_async(prompt=prompt)) # type: ignore" | ||
| "print(\"CodeChameleon:\", await code_chameleon.convert_async(prompt=prompt)) # type: ignore\n", | ||
| "\n", | ||
| "# PUZZLED [@ahn2025puzzled] hides sensitive words in a word puzzle the target must solve\n", |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
PuzzledConverter, which implements the PUZZLED jailbreak from "PUZZLED: Jailbreaking LLMs through Word-Based Puzzles" (Ahn & Lee, arXiv:2508.01306).Closes #2234. @romanlutz confirmed on the issue that a converter is the right shape for this.
The idea from the paper: instead of asking a model something harmful directly, you hide the sensitive words of the request inside a word puzzle and ask the model to solve the puzzle first, then follow the instruction it just rebuilt. The harmful wording never appears in plain text.
How the converter works:
random.Randomyou can seed.converter_target, it also asks that model for the paper's indirect "hint" clue. If that call fails or returns junk, it falls back to the plain clue.It uses the standard
Converterinterface, so it works with any target, scorer, or other converters.Tests and Documentation
60 unit tests in
tests/unit/converter/cover word selection and masking, all three puzzle builders, and the converter itself (including the optional LLM-clue path and its fallbacks). They're deterministic and don't touch the network.Docs: added a usage example to
doc/code/converters/1_text_to_text_converters.pyand the citation todoc/references.bibanddoc/bibliography.md. The converter cell runs offline; I didn't regenerate the paired.ipynbbecause other cells in that notebook need live model endpoints.