[ValentinTorassa]

Fixes #92

What changed

parseAgentJsonArray now classifies hard provider policy refusals (Anthropic
AUP blocks, OpenAI safety-system rejections, content_policy_violation)
before attempting jsonrepair, and throws a dedicated
AgentPolicyRefusalError. All six investigate/revalidate parse sites across
the three agent backends rethrow it immediately instead of sending the
JSON-repair follow-up.

Why

When a provider refuses on policy grounds, the refusal text currently reaches
the findings parser and fails as "wasn't a parseable JSON findings array" —
reported as a formatting problem when the real condition is zero coverage
over the batch.

Worse, the parse-failure path then sends buildInvestigateJsonRepairPrompt
to the same session. That prompt asks the model to "re-output the same
conclusions" and explicitly accepts an empty findings array for files with no
issues. If a model that just declined complies, the batch parses cleanly and
those files are recorded as investigated-with-no-findings — a silent coverage
gap, indistinguishable from a clean batch. With models that decline security
work as a documented behavior (e.g. Fable, per the DeepsecBench post), this
is an expected operating condition, not an edge case.

Unlike QuotaExhaustedError, this stays per-batch: a refusal depends on
batch content, so the rest of the run continues and a rerun picks the failed
batch up.

Verification

• pnpm test passes (added regression tests: refusal text →
  AgentPolicyRefusalError from both parseInvestigateResults and
  parseRevalidateVerdicts; valid findings JSON quoting refusal text as
  evidence still parses normally)
• pnpm lint passes
• pnpm knip passes
• If this adds a matcher: n/a

Also ran pnpm -r build, pnpm test:bundle, and pnpm typecheck:deepsec —
all green.

Notes for reviewer

• Classifier is strict, not loose — the opposite trade-off from
  classifyQuotaError: a false positive would relabel ordinary malformed JSON
  as zero-coverage, which is worse than the status quo. Only explicit
  provider-refusal strings match; anything else degrades to the existing
  fail-loud parse error.
• Classification happens inside the JSON.parse catch, so valid findings
  JSON that merely quotes refusal text (e.g. as evidence in a description)
  parses normally — covered by a regression test.
• Per-batch, not run-aborting — happy to switch to abort-the-run (like
  quota) if you'd rather treat a hard refusal as a stop condition.
• Out of scope, possible follow-ups: surfacing refusals as their own
  category in the end-of-run summary (they currently collapse into
  "N batch(es) failed"), and classifying refusals that arrive as SDK error
  results rather than result text (that path already fails loud, just with a
  less precise message).

Found this while preparing a hands-on review of deepsec for my
security-engineering YouTube channel
(@vtcibersecurity, dev/appsec
audience) — happy to adjust anything here, and to take on the
summary-category follow-up if you want it.

[vercel]

@ValenTorassaColomberoTeramot is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

[ValentinTorassa]

Merged main into the branch to clear the conflicts that came with #139. They were only the import lists of the three agent files and of shared.test.ts; the change itself is the same.

Checked locally on the merged branch: pnpm -r build, biome check and the full vitest run with the bundle built (2,425 tests pass).

[auto-maintain]

🤖 auto-maintain review

Automated, advisory triage for @ValentinTorassa's PR. Facts below are read from the GitHub API.

Check	Result
Author's merged PRs (this repo)	0 — ⚠️ first-time contributor
Account established	✅ (age 1165d · 371 followers · 28 public repos)
Commits signed/verified	⚠️ 0/2

Review panel: 🟡 medium highest severity

deepsec maintainer code review: 🟡 medium

The refusal classifier's placement both misses structured refusal envelopes and creates false refusals on malformed findings.

• packages/processor/src/agents/shared.ts:815 — Valid JSON refusal envelopes such as {"error":{"code":"invalid_prompt"}} bypass classification because it only runs when JSON.parse throws; agents still send the misleading JSON-repair follow-up.
• packages/processor/src/agents/shared.ts:820 — Repairable malformed findings that mention invalid_prompt, content_policy_violation, or the AUP URL are misclassified as provider refusals before jsonrepair runs, dropping legitimate findings.

General code review: 🟡 medium

Valid JSON policy-refusal envelopes remain unclassified, leaving a core failure path unfixed.

• packages/processor/src/agents/shared.ts:815 — Policy classification only runs when JSON.parse throws. Valid provider envelopes such as {"error":{"code":"invalid_prompt"}} and content_policy_violation parse successfully, fall through as non-array output, and still trigger the unsafe JSON-repair follow-up this change intends to prevent.

Scanner and matcher quality: 🟡 medium

The refusal classifier has both a concrete envelope-detection gap and a false-positive path that can undermine batch coverage.

• packages/processor/src/agents/shared.ts:815 — Valid JSON refusal envelopes such as {"error":{"code":"invalid_prompt"}} bypass classification because it runs only in JSON.parse's catch; they trigger the JSON-repair follow-up and can still become a silent empty batch.
• packages/processor/src/agents/shared.ts:257 — Matching bare `invalid_prompt` or `content_policy_violation` anywhere in malformed output misclassifies repairable findings that quote those source-code tokens, discarding real findings as a provider refusal.

Sandbox and credential boundaries: 🟡 medium

Refusal classification and the six rethrow sites are structurally correct (cleanup/dispose preserved, failures stay loud and per-batch), but matching over the full result text lets untrusted scanned-repo content trigger a false refusal and drop a batch's findings.

• packages/processor/src/agents/shared.ts:820 — `classifyPolicyRefusal` is run against the entire `resultText`, including the fenced JSON payload, which routinely contains verbatim repository content (finding `snippet`/description evidence). A batch whose JSON is merely malformed (the case jsonrepair exists for) but whose snippets quote a scanned file containing `invalid_prompt`, `content_policy_violation`, or the AUP URL is now discarded as a provider refusal instead of being repaired — the batch's real findings are lost and mislabeled as zero-coverage. Anyone who can add a string to a scanned file (i.e. the untrusted PR under scan) can influence this; classify only the text outside the extracted JSON payload, or only when no array candidate is present.
• packages/processor/src/agents/shared.ts:257 — `/\binvalid_prompt\b/` is a bare identifier token, not a provider error envelope — it matches any occurrence anywhere in agent output, e.g. an echoed symbol name or key from scanned source. This is the weakest of the three markers and contradicts the "only explicit provider-refusal strings match" claim; require the surrounding error shape (e.g. `"code":"invalid_prompt"`) instead.

Durability and distributed state: 🟡 medium

The refusal handling misses structured JSON refusals and can falsely reject repairable findings output.

• packages/processor/src/agents/shared.ts:848 — Valid JSON refusal envelopes bypass classification: JSON.parse succeeds for `{"error":{"code":"invalid_prompt"}}`, then the non-array error triggers the JSON-repair follow-up, which can still convert a refused batch into silent zero coverage.
• packages/processor/src/agents/shared.ts:820 — Classification scans all malformed output for bare tokens such as `content_policy_violation`; a repairable findings array mentioning that token in a finding is misclassified as a refusal, repeatedly leaving the batch errored instead of applying jsonrepair.

_{Posted by auto-maintain. This automated code review is advisory; a human maintainer makes the call.}

[ValentinTorassa]

Gentle follow-up: the branch is still up to date with main and mergeable, and the build, biome check and the full vitest run pass (details in my previous comment). The two failing Vercel checks are only the fork deploy waiting for team authorization. Happy to adjust scope, naming or tests if that helps the review.