Skip to content

feat(setup): configure other models as adversarial reviewers, with per-harness commands - #1371

Merged
potiuk merged 5 commits into
mainfrom
feat/adversarial-review-setup
Sep 24, 2026
Merged

potiuk merged 5 commits into
mainfrom
feat/adversarial-review-setup

Conversation

@potiuk

@potiuk potiuk commented Sep 24, 2026

Copy link
Copy Markdown
Member

PR 2 of 4. Follows #1368 (merged), which added the tool and plugin. PR 3 and PR 4 stack on this one.

Summary

  • Per-harness commands. adversarial-review commands --harness claude|codex|gemini|copilot prints each harness's command file from one template:
    • The Claude Code command ships in the plugin as /magpie-adversarial-review:adversarial-review.
    • Codex and Gemini get user-level files: ~/.codex/prompts/… and ~/.gemini/commands/….
    • Copilot gets the one-line invocation.
    • No command bakes a plugin version in; a plugin upgrade leaves them valid.
  • setup config adversarial-review (Step 3c). It runs detect, pre-ticks every available model CLI except the harness's own, and writes .apache-magpie-local/adversarial-review.md. It then offers the command files. It runs only when named, never from a skill's pre-flight. Home-directory writes are the one, named exception to config's write boundary.
  • verify 8i flags configured reviewers whose CLI has gone. adopt 4a always treats the file as personal.
  • Sandbox exclusion. One excludedCommands entry names the installed plugin's single-line invocation, and the plugin cache is Edit-denied. There is deliberately no allow: each run sends the change to other model providers and keeps its prompt. The settings baseline and expected.json change in lockstep. isolated-setup-install Step R wires the entry, and isolated-setup-verify check 14 verifies it.

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Python package (tools/*/ with pyproject.toml)
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)

Test plan

  • prek run --all-files passes
  • For Python packages touched: pytest (140 tests), ruff check and mypy pass for tools/adversarial-review
  • For skill behaviour changes: new eval suite setup/step-config-adversarial (5 cases). It covers the named run, a plain run, a pre-flight entry, a missing plugin, and only self available.
  • Other:
    • A test renders every harness's command and checks the invocation against the adversarial-review exclusion pattern in tools/sandbox-lint/expected.json. A quoted or ~-expanded path would silently stay sandboxed.
    • The shipped Claude Code command is pinned byte-for-byte to its generator.
    • The config template is pinned as parseable.
    • sandbox-lint passes (settings and baseline match).
    • The tool refuses a --body-file or diff: path outside the repository or a temporary directory, so an approved run cannot send ~/.ssh to a model.

RFC-AI-0004 compliance

  • HITL: config Step 3c proposes and writes only what the user ticks. The tool has no allow rule, so every run keeps its permission prompt.
  • Sandbox: one narrow exclusion, for the single-line plugin invocation only, with an Edit deny on the plugin cache. The residual glob caveat is documented in Step R.
  • Vendor neutrality: commands for four harnesses from one template.
  • Conversational + correctable: configuration lives in adversarial-review.md (personal, with an optional project default).

Linked issues

Follows #1368.

Notes for reviewers (optional)

  • Exclusion form: the exclusion only matches the command spelled with a literal ~ and an unquoted path. Every place that tells an agent how to run the tool now uses that exact form.
  • Codex: Codex's default workspace-write sandbox has no network, so the Codex and Gemini commands tell the agent to ask for the one command to run outside its sandbox.

🤖 Generated with Claude Code

@potiuk potiuk added family:setup setup-* skills capability:platform Framework / agent substrate skills (install, verify, doctor, override, status, setup bootstrap) labels Sep 24, 2026
@potiuk
potiuk added this pull request to stack #1374 September 24, 2026 16:52
…ommand in the plugin

`adversarial-review commands --harness claude|codex|gemini|copilot` prints the
command file for each harness from one template. The Claude Code command is
generated into tools/adversarial-review/commands/ (pinned by a test against
the generator) and published by the plugin as
/magpie-adversarial-review:adversarial-review. Harness command files are kept
out of doctoc: their frontmatter must come first.

Generated-by: Claude Opus 5
…y, upgrade

- config Step 3c: run `adversarial-review detect`, pre-tick every available
  backend except the harness's own model, write
  .apache-magpie-local/adversarial-review.md, and offer the Codex/Gemini
  command files under the user's home (never on a pre-flight entry, never
  inside a repository; hard rule 1 names the exception)
- verify 8i: configured reviewers whose CLI is gone, and stale command files
- upgrade 6b: rewrite the command files when the plugin path changes
- adopt 4a: adversarial-review.md is personal by default
- projects/_template/adversarial-review.md, pinned parseable by a test
- eval suite setup/step-config-adversarial (4 cases)

Generated-by: Claude Opus 5
The reviewer CLIs need network and their own credentials, which the sandbox
denies. One excludedCommands entry names the installed plugin's single-line
invocation; the plugin cache is Edit-denied, since the tool runs unsandboxed;
and there is deliberately no allow rule, so every run keeps its prompt.
isolated-setup-install Step R wires it; the settings baseline and the
secure-setup reference change in lockstep.

Generated-by: Claude Opus 5
- one invocation form everywhere — unquoted, literal ~, <version> — the form
  the sandbox exclusion matches, pinned by a test against the baseline; no
  command bakes a version in, so the upgrade rewrite is gone
- Codex/Gemini/Copilot commands tell the agent to ask for an unsandboxed run
- config Step 3c runs only when named, diffs an existing file, and the
  recap lists the home-directory writes; the argument is documented
- verify 8i also runs on marketplace installs; adopt always flags the file
  as personal and says what promoting on-pr-create costs contributors
- isolated-setup-verify check 14 for the exclusion, the deny, and no allow
- the tool refuses --body-file / diff: paths outside the repo or temp dir

Generated-by: Claude Opus 5
Generated-by: Claude Opus 5
@potiuk
potiuk force-pushed the feat/adversarial-review-setup branch from a33e9a0 to 290a640 Compare September 24, 2026 17:05
@potiuk
potiuk merged commit cb29fc5 into main Sep 24, 2026
12 checks passed
@potiuk
potiuk deleted the feat/adversarial-review-setup branch September 24, 2026 17:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capability:platform Framework / agent substrate skills (install, verify, doctor, override, status, setup bootstrap) family:setup setup-* skills

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant