[rwagwani]

Stacked on #306. Please review that one first; this PR's own changes are the last two commits, 8e8d237 and c11dc88.

Split out of #306 at @lfengad's suggestion. The commits here also fix the span bug found in that review.

The problem

The matcher fuses neighbouring words. That is what catches shot gun as the entry shotgun, but it also turns ordinary prose into an entry: a robot sitting at a desk in the background is blocked as deskin. The matcher's whitelist only holds single tokens, and whitelisting either half cannot help, because fusing does not consult the whitelist.

What changes

• A whitelist entry with a space in it is a phrase. A prompt is cleared only if every match lies entirely inside an occurrence of a whitelisted phrase. Matches that straddle a phrase boundary, and matches anywhere else in the prompt, keep it blocked.
• Phrases are located before invisible characters are folded. So desk<U+200B>in, which renders as the entry, cannot reach the exemption.
• With no phrase in the prompt, the result and the message are exactly what they were. This has its own early return. The production whitelist has no phrases today, so the mechanism is inert until list owners add one.
• Load-time warning: a whitelisted word that the matcher censors anyway, as another entry's leetspeak variant (flat from fiat), is now logged.

Review follow-up: real match spans

The previous version rebuilt each match's span from the censored output, by counting tokens. Thank you for spotting that this is ambiguous. In the example from the review, the blocklist is nicecamera and shotgun, the whitelist is nice camera shot and gun show, and the prompt is nice camera shot gun show. The guess split it as nice camera shot + gun, both inside a phrase, so the prompt was allowed.

Fixing it turned up two more cases of the same kind, both fixed here:

• Greedy matching: the matcher is greedy, so with deskin and in gun both listed, a desk in gun only ever reports desk in. The in gun match never appears in the censored output. The old "second occurrence" test passed only because the miscounted span happened to block it.
• One-letter final word: better_profanity ignores a one-letter word at the very end of its input. So a span such as batma n, plan b or a one-letter entry, censored on its own, matched nothing, and a whitelisted phrase elsewhere in the prompt cleared it (snow white batma n more with snow white whitelisted). An independent review pass found this one.

How it works now:

• fused_matches() tries every span on its own, with a word appended after it so the matcher reads the span as it would inside a prompt. A span counts as a match when the matcher reduces it to one censored token.
• The width is bounded by the matcher's own MAX_NUMBER_COMBINATIONS.
• The exemption is decided per match.
• Fail closed: if the prompt as a whole was censored but no span on its own matches, it stays blocked.

Results:

Check	Result
Your example	blocked
Adjacent, overlapping, three-in-a-row, punctuation, zero-width, fullwidth and one-letter-ending cases	blocked (parametrized tests through is_safe)
Repeated exempt phrases (a desk in, a desk in the corner)	allowed. The old count-based spans blocked them.
Old span code against the new tests	the new tests fail
Removing both the appended word and the fail-closed guard	the one-letter tests fail; either guard alone makes them pass
Randomized checks on the final code, 160,000 prompts in total (see below)	0 wrong passes
The same randomized checks on the previous head	they find wrong passes

The randomized checks:

• Three generators:
  • a collision-heavy vocabulary (desk in shot gun nice camera show bat man …, with punctuation and case);
  • the same vocabulary with one-letter words, leetspeak and hyphenated tokens added;
  • an unrelated phrase followed by random tails.
• Compared against main and a brute-force reference that knows nothing about the span code.
• They confirm three things:
  1. Without phrases, the result equals main.
  2. With phrases, nothing is blocked that main allows.
  3. Whenever a phrase clears a prompt that main blocks, every occurrence of an entry that the matcher recognises lies inside a phrase.

These checks cover the shapes in those vocabularies; they are not a proof.

Cost

The fused path runs only when a prompt contains both a whitelisted phrase and a match. To bound it:

• a 32-token chunk that the matcher clears in one call is skipped;
• results are cached across the readings is_safe checks;
• past 4,000 matcher calls the prompt stays blocked, as it is without phrase support.

Measured with the shipped lists plus a desk in phrase (main is 0.01–0.13 s on all of these):

Prompt	This PR
300-token ordinary prompt with one phrase	about 0.7–0.9 s
800 tokens of repeated phrase + a non-ASCII character	2.1 s (was 25.5 s before the bound)
800 tokens of distinct split entries	1.7 s

Known limits

• The exempted spelling itself passes. That is inherent to clearing the false positive.
• The phrase also clears its lemmatized forms (desks in), because the exemption applies on the lemmatized pass too.
• Punctuation inside the phrase is not exempted (a desk, in the corner stays blocked). This errs strict.
• No phrase is added to the production whitelist here. Adding desk in is a data change for the list owners.

Tests

• pytest cosmos_framework/auxiliary/guardrail/blocklist/: 107 passed.
• ruff and pre-commit (both CI configs, all files) are clean.
• With the shipped lists, the 1459-prompt corpus and the 6-spelling sweep match fix(guardrail): fold fullwidth, accented, lookalike and invisible spellings before to_ascii erases them #306 exactly.

🤖 Generated with Claude Code