Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .claude/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"gh *",
"uv run --project tools/vetted-ops vetted-op-read *",
"uv run --project ~/.claude/plugins/cache/apache-magpie/magpie-vetted-ops/*/tools/vetted-ops vetted-op-read *",
"uvx --from ~/.claude/plugins/cache/apache-magpie/magpie-adversarial-review/*/tools/adversarial-review adversarial-review *",
"~/.claude/scripts/magpie-run-evals.sh *"
],
"filesystem": {
Expand Down Expand Up @@ -126,6 +127,7 @@
"Edit(.claude/settings.local.json)",
"Edit(.apache-magpie-overrides/tools/vetted-ops/**)",
"Edit(~/.claude/plugins/cache/apache-magpie/magpie-vetted-ops/**)",
"Edit(~/.claude/plugins/cache/apache-magpie/magpie-adversarial-review/**)",
"Edit(~/.claude/scripts/**)",
"Bash(curl *)",
"Bash(wget *)",
Expand Down
6 changes: 5 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,11 @@ repos:
# as the skill definitions above. The block source was not excluded
# before and did carry a TOC, which rode into all 65 propagated
# copies as a table of contents for a file none of them are.
exclude: ^(\.claude/skills/.*|\.agents/skills/.*|\.github/skills/.*|skills/.*|plugins/magpie-[^/]+/skills/.*|tools/cve-tool-vulnogram/generate-cve-json/SKILL\.md|tools/skill-evals/.*|tools/spec-loop/.*|\.github/PULL_REQUEST_TEMPLATE\.md|tools/dev/blocks/.*|tools/dev/preflight-block\.md|tools/setup-preflight/src/setup_preflight/sections/.*)$
# Skip harness command files (plugins/magpie-*/commands/ and the
# generated source in tools/adversarial-review/commands/): frontmatter
# first, like skills, and the source is pinned byte-for-byte to its
# generator by a test, which a TOC wrapper would break.
exclude: ^(plugins/magpie-[^/]+/commands/.*|tools/adversarial-review/commands/.*|\.claude/skills/.*|\.agents/skills/.*|\.github/skills/.*|skills/.*|plugins/magpie-[^/]+/skills/.*|tools/cve-tool-vulnogram/generate-cve-json/SKILL\.md|tools/skill-evals/.*|tools/spec-loop/.*|\.github/PULL_REQUEST_TEMPLATE\.md|tools/dev/blocks/.*|tools/dev/preflight-block\.md|tools/setup-preflight/src/setup_preflight/sections/.*)$
args:
- "--maxlevel"
- "3"
Expand Down
15 changes: 15 additions & 0 deletions docs/designs/2026-09-23-adversarial-review-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
- [Task 2.3: Configuration template and `setup config`](#task-23-configuration-template-and-setup-config)
- [Task 2.4: `setup verify` and `setup adopt`](#task-24-setup-verify-and-setup-adopt)
- [Task 2.5: Sandbox exclusion](#task-25-sandbox-exclusion)
- [PR 2 as built, after its whole-branch review](#pr-2-as-built-after-its-whole-branch-review)
- [PR 3 — the shared pre-PR block in every PR-creating skill](#pr-3--the-shared-pre-pr-block-in-every-pr-creating-skill)
- [Task 3.1: Block source](#task-31-block-source)
- [Task 3.2: Declare the region in each PR-creating skill](#task-32-declare-the-region-in-each-pr-creating-skill)
Expand Down Expand Up @@ -2648,6 +2649,20 @@ Task-level. Each task follows the same TDD loop as PR 1.
- **Behaviour:** add `uvx --from ~/.claude/plugins/cache/apache-magpie/magpie-adversarial-review/*/tools/adversarial-review adversarial-review *` as the one `excludedCommands` entry, mirroring vetted-ops. Document that a compound command falls back into the sandbox.
- **Tests:** sandbox-lint accepts the entry and rejects a broader `uvx *`.

### PR 2 as built, after its whole-branch review

- **One command form everywhere:** `uvx --from ~/.claude/plugins/cache/apache-magpie/magpie-adversarial-review/<version>/tools/adversarial-review …`, unquoted with a literal `~`, which is the form the sandbox exclusion matches.
- A test checks every generated command against the pattern in `tools/sandbox-lint/expected.json`.
- Claude Code's command reads `<version>` from `${CLAUDE_PLUGIN_ROOT}`; the others resolve the newest installed version at run time.
- No command bakes a version in, so the `upgrade` rewrite this plan called for is not needed and was dropped. The `commands` flag is `--plugin-dir` (optional), not `--plugin-root`.
- **Gemini command:** it lives at `~/.gemini/commands/`, and the agent runs the tool through its own shell tool, not through `!{…}` injection.
- **Sandbox hint:** the Codex, Gemini and Copilot commands tell the agent to ask for the one command to run outside its sandbox.
- **`config` Step 3c** runs only when named (`config adversarial-review`). A plain `config` run mentions it in the recap, a pre-flight entry never touches it, and an existing file is shown as a diff before it is replaced.
- **`verify` 8i** is also run on marketplace installs. `adopt` 4a always flags `adversarial-review.md` as personal.
- **Canonical secure-setup list:** check 14 covers the exclusion, the deny and the absence of an `allow`, in `isolated-setup-verify`.
- **Input paths:** the tool refuses a `--body-file` or `diff:` path outside the repository or a temporary directory.
- **Not built:** a unit test for `verify` 8i and an `adopt` eval (both prose steps), and a `sandbox-lint` rule rejecting a broad `uvx *`. `sandbox-lint` compares against its baseline rather than judging entries, so the baseline pairs the settings file and review guards it.

---

## PR 3 — the shared pre-PR block in every PR-creating skill
Expand Down
6 changes: 3 additions & 3 deletions docs/designs/2026-09-23-adversarial-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@

| | |
|---|---|
| **Status** | Being built: the tool and plugin (PR 1 of 4). Plan: [`2026-09-23-adversarial-review-plan.md`](2026-09-23-adversarial-review-plan.md). |
| **Status** | Being built: the tool and plugin (PR 1 of 4, apache/magpie#1368), then `setup` (PR 2 of 4). Plan: [`2026-09-23-adversarial-review-plan.md`](2026-09-23-adversarial-review-plan.md). |
| **Scope** | A new tool (`tools/adversarial-review`) and substrate plugin, the `setup` family (detection, configuration, per-harness commands), a shared pre-PR block included by every PR-creating skill, and an optional multi-reviewer second read in `pr-management-code-review`. |

## What is wrong
Expand Down Expand Up @@ -124,9 +124,9 @@ adversarial_review:

| Harness | Command |
|---|---|
| Claude Code | `/magpie-adversarial-review`, shipped in the plugin |
| Claude Code | `/magpie-adversarial-review:adversarial-review`, shipped in the plugin (Claude Code namespaces plugin commands by plugin name) |
| Codex CLI | `~/.codex/prompts/magpie-adversarial-review.md` |
| Gemini CLI | `.gemini/commands/magpie-adversarial-review.toml` |
| Gemini CLI | `~/.gemini/commands/magpie-adversarial-review.toml` |
| Copilot CLI | a reusable-prompt entry if the installed Copilot CLI supports one; otherwise `setup` prints the one-line `adversarial-review run` invocation to use instead |

Where a harness has no user-defined command mechanism, the fallback is always the same single-line tool invocation, which works from any harness's shell.
Expand Down
2 changes: 1 addition & 1 deletion docs/designs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ what was designed and deliberately not built.
| [Body-owned configuration layers](2026-09-17-body-owned-config-layers.md) | Proposed — depends on the Incubator PMC and ComDev |
| [Reproducible releases](2026-09-20-reproducible-releases.md) | Built (apache/magpie#1296); the ASF automated-signing path and the ATR SWHID comparison await first use |
| [Reconciliation tracking for marketplace installs](2026-09-21-marketplace-reconciliation-tracking.md) | Built, bar seven items it names |
| [Adversarial review by other models, before every PR](2026-09-23-adversarial-review.md) | Being built — PR 1 of 4 (the tool and plugin); [plan](2026-09-23-adversarial-review-plan.md) |
| [Adversarial review by other models, before every PR](2026-09-23-adversarial-review.md) | Being built — the tool and plugin (apache/magpie#1368), then `setup` (PR 2 of 4); [plan](2026-09-23-adversarial-review-plan.md) |

One document per subject, describing the result rather than the phases it was
built in. While a design is being implemented it may be split into plans; when
Expand Down
8 changes: 4 additions & 4 deletions docs/mode-economics.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ special-token spellings counted as ordinary text.
Coverage: **75 of 75 local `skills/*/SKILL.md` files**.
External `source.md` redirects and harness symlinks are excluded.

Measurement manifest SHA-256: `622444fcab63b8fc56d6741c5d7aca88c13468422e2122df1e915573d7e1f852`.
Measurement manifest SHA-256: `9a53dff1a0bfd28472c46c4b0e4b6c060a99736672c6a7816a409331ddbe5fb8`.

| Skill file | Measured tokens | Source SHA-256 (first 16 characters) |
|---|---:|---|
Expand Down Expand Up @@ -188,11 +188,11 @@ Measurement manifest SHA-256: `622444fcab63b8fc56d6741c5d7aca88c13468422e2122df1
| [security-model-update](../skills/security-model-update/SKILL.md) | 4,842 | `7b3e5fda39ea64a6` |
| [security-model-verify](../skills/security-model-verify/SKILL.md) | 5,541 | `17170f3e573564b8` |
| [security-tracker-stats-dashboard](../skills/security-tracker-stats-dashboard/SKILL.md) | 3,815 | `8bf5804a8f214d6a` |
| [setup](../skills/setup/SKILL.md) | 4,185 | `751ad44b2bd50e11` |
| [setup](../skills/setup/SKILL.md) | 4,229 | `9d1f0c2cbeda3422` |
| [setup-isolated-setup-doctor](../skills/setup-isolated-setup-doctor/SKILL.md) | 5,539 | `ba5d0773615f463d` |
| [setup-isolated-setup-install](../skills/setup-isolated-setup-install/SKILL.md) | 3,875 | `71c657ea2aaa48c4` |
| [setup-isolated-setup-install](../skills/setup-isolated-setup-install/SKILL.md) | 4,399 | `9b8ed9a8354e2f60` |
| [setup-isolated-setup-update](../skills/setup-isolated-setup-update/SKILL.md) | 4,013 | `8f851263948c42a3` |
| [setup-isolated-setup-verify](../skills/setup-isolated-setup-verify/SKILL.md) | 4,691 | `fa9ec75b26714c97` |
| [setup-isolated-setup-verify](../skills/setup-isolated-setup-verify/SKILL.md) | 4,764 | `71a193ecf4cc2f15` |
| [setup-override-upstream](../skills/setup-override-upstream/SKILL.md) | 3,677 | `48dbbbf633297062` |
| [setup-privacy-llm](../skills/setup-privacy-llm/SKILL.md) | 2,051 | `daf38d5849397854` |
| [setup-shared-config-sync](../skills/setup-shared-config-sync/SKILL.md) | 3,833 | `8a53b61ee3d3cb56` |
Expand Down
26 changes: 24 additions & 2 deletions docs/setup/secure-agent-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -490,7 +490,15 @@ below, annotated.
// with `x509: OSStatus -26276`. Details, the `gh tofile` alias
// workaround, and the upstream report (anthropics/claude-code#95532)
// are in sandbox-troubleshooting.md → "`gh` fails with TLS …".
"excludedCommands": ["gh *"],
// The adversarial-review tool runs other models' CLIs, which need network
// access and their own credentials (~/.codex, ~/.copilot, ~/.gemini,
// ~/.claude). Only its single-line, installed-plugin form is excluded;
// it keeps its permission prompt (no `allow`), and the plugin cache is
// `Edit`-denied below. See the isolated-setup-install skill, Step R.
"excludedCommands": [
"gh *",
"uvx --from ~/.claude/plugins/cache/apache-magpie/magpie-adversarial-review/*/tools/adversarial-review adversarial-review *"
],
// The `lychee` link-check hook runs in OFFLINE mode (`offline =
// true` in `.lychee.toml`): it validates only local cross-file and
// anchor references and never fetches remote URLs, so it makes no
Expand Down Expand Up @@ -650,7 +658,10 @@ below, annotated.
// every file-writing tool (Write and NotebookEdit included), and a
// `Write(path)` rule is not matched by the file permission check at all.
"Edit(~/.claude/plugins/cache/apache-magpie/magpie-vetted-ops/**)",
"Edit(.apache-magpie-overrides/tools/vetted-ops/**)"
"Edit(.apache-magpie-overrides/tools/vetted-ops/**)",
// The adversarial-review tool runs unsandboxed (excludedCommands above), so
// the code it runs must not be editable by the agent that calls it.
"Edit(~/.claude/plugins/cache/apache-magpie/magpie-adversarial-review/**)"
],
"ask": [
"Bash(git push *)", // including --force / --force-with-lease variants
Expand Down Expand Up @@ -3212,6 +3223,17 @@ below and report ✓ done / ✗ missing / ⚠ partial, with the evidence
be agent-writable. Copies that differ from
`tools/skill-evals/` are ⚠, not ✗: the harness runs, it just
grades against an older runner than the tree's.
14. **Adversarial-review exclusion**, if the
`magpie-adversarial-review` plugin is installed (n/a otherwise).
`sandbox.excludedCommands` contains
`"uvx --from ~/.claude/plugins/cache/apache-magpie/magpie-adversarial-review/*/tools/adversarial-review adversarial-review *"`,
`permissions.deny` contains
`Edit(~/.claude/plugins/cache/apache-magpie/magpie-adversarial-review/**)`,
and **no** `permissions.allow` entry covers the tool. A missing
exclusion is ⚠ (every reviewer reports `unavailable` from inside
the sandbox); a missing deny is ✗ (the exclusion runs that code
unsandboxed); an `allow` is ✗ (each run sends the change to other
model providers and must keep its prompt).
```

Re-run either form after every Claude Code upgrade — the sandbox
Expand Down
48 changes: 47 additions & 1 deletion plugins/magpie-setup/skills/isolated-setup-install/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ when_to_use: >-
If it is already installed, use `setup-isolated-setup-verify` to check
it or `setup-isolated-setup-update` to refresh it.
capability: capability:platform
surface_hash: sha256:e78e03834f4fdec8
surface_hash: sha256:eb1b228a501f2772
license: Apache-2.0
---

Expand Down Expand Up @@ -269,6 +269,52 @@ least-privilege hygiene for a cooperating skill, and it is worth
having for that, but it stops nothing that chooses to name a
different caller.

### Step R — The adversarial-review exclusion

Only applies when the `magpie-adversarial-review` plugin is installed. If it
is not, skip this step and say so.

The reviewer CLIs it runs (`codex`, `copilot`, `gemini`, `claude`) need
network access and read their own credentials (`~/.codex`, `~/.copilot`,
`~/.gemini`, `~/.claude`), which this sandbox denies. The tool therefore
runs outside it, through one exclusion that names the installed plugin:

```jsonc
"sandbox": {
"excludedCommands": [
"uvx --from ~/.claude/plugins/cache/apache-magpie/magpie-adversarial-review/*/tools/adversarial-review adversarial-review *"
]
},
"permissions": {
"deny": [
// the tool runs unsandboxed, so the code it runs must not be editable
// by the agent that calls it
"Edit(~/.claude/plugins/cache/apache-magpie/magpie-adversarial-review/**)"
]
}
```

**Do not add an `allow` for it.** Every run sends the change to other
model providers and costs money; the harness prompt is the gate, by design.

Tell the operator what the exclusion covers and what it does not:

- It matches only the single-line form, spelled with a literal `~` and an
unquoted path. A pipe, `$(…)`, `&&`, a redirection, quotes, or an
expanded home directory put the command back in the sandbox, where the
reviewer CLIs fail to read their credentials and report `unavailable`.
- The `*` in the pattern can match more than a version directory — a path
with `..` segments would still match. The prompt that every run keeps is
the gate against that; read the path in it before approving.
- The tool refuses a `--body-file` or `diff:` file outside the repository
or a temporary directory, so an approved run cannot be pointed at
`~/.ssh` or a private checkout to send it to a model.
- Outside the sandbox the tool only runs each reviewer CLI in its own
read-only mode, and writes nothing to the repository. `codex`'s
read-only mode still reads files anywhere on the machine; see the tool's
README for what that means for a machine that also holds a private
checkout.

### Steps K, L and M — optional extras

None of these is needed for a working install. Walk the one the operator
Expand Down
7 changes: 6 additions & 1 deletion plugins/magpie-setup/skills/isolated-setup-verify/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ when_to_use: >-
have started working — that is the canary for a denial having
silently become an allow.
capability: capability:platform
surface_hash: sha256:b3582f9511e28ef0
surface_hash: sha256:2321708548dbd769
license: Apache-2.0
---

Expand Down Expand Up @@ -184,6 +184,11 @@ Walk each in order:
Otherwise:
[`conditional-checks.md`](conditional-checks.md#check-13--eval-harness-exclusion-if-installed).

14. **Adversarial-review exclusion, if installed.** Optional (step R of `setup-isolated-setup-install`).
**n/a** when the `magpie-adversarial-review` plugin is not installed.
Otherwise:
[`conditional-checks.md`](conditional-checks.md#check-14--adversarial-review-exclusion-if-installed).

## After the report

If every check is ✓, say so explicitly and stop; no further suggestion is needed.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -301,3 +301,23 @@ reading this file; read the section for a check whose condition holds.

Rationale:
[`tools/skill-evals/README.md` → Running from inside the sandbox](../../../../tools/skill-evals/README.md#running-from-inside-the-sandbox).

## Check 14 — adversarial-review exclusion, if installed

14. **Adversarial-review exclusion, if installed.** Optional (step R of
`setup-isolated-setup-install`): report **n/a** when the
`magpie-adversarial-review` plugin is not installed.

When it is:

- **14a — the exclusion.** `sandbox.excludedCommands` contains
`uvx --from ~/.claude/plugins/cache/apache-magpie/magpie-adversarial-review/*/tools/adversarial-review adversarial-review *`.
Missing is ⚠, not ✗: nothing unsafe happens, but every run stays
sandboxed, where the reviewer CLIs cannot read their credentials,
and every reviewer reports `unavailable`.
- **14b — the plugin cache is not agent-writable.** `permissions.deny`
contains `Edit(~/.claude/plugins/cache/apache-magpie/magpie-adversarial-review/**)`.
Missing is ✗: the exclusion runs that code outside the sandbox.
- **14c — no `allow`.** No `permissions.allow` entry matches the tool's
invocation. One is ✗: each run sends the change to other model
providers and must keep its prompt.
3 changes: 2 additions & 1 deletion plugins/magpie-setup/skills/setup/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ when_to_use: >-
contributor and is not an install.
argument-hint: "[install|config|adopt|unadopt|upgrade|worktree-init|verify|reconcile|override skill-name|uninstall]"
capability: capability:platform
surface_hash: sha256:eb6962f782b4b45a
surface_hash: sha256:3bc8ddeb317a6377
license: Apache-2.0
---

Expand Down Expand Up @@ -133,6 +133,7 @@ The skill dispatches by the first positional argument:
| `setup install method:svn-zip\|git-tag\|git-branch` | [`install.md`](install.md) | The pinned snapshot install — the fallback path. Main-checkout only. |
| `setup config` | [`config.md`](config.md) | **Not an install, and not adoption.** Configure the installed skills for *you*, in gitignored `.apache-magpie-local/`. Works on any repo, adopted or not, with nobody's permission. Writes nothing committable and stages nothing. |
| `setup config <skill>` | [`config.md`](config.md) | The same, narrowed to one skill's required configuration. |
| `setup config adversarial-review` | [`config.md`](config.md#step-3c--adversarial-reviewers-optional) | Detect the installed model CLIs and configure them as adversarial reviewers. |
| `setup adopt` | [`adopt.md`](adopt.md) | **Not an install.** Commit the repo's recommended default plugin set and scaffold its overrides store, so every contributor arrives with them. Needs an explicit maintainer decision; stages, never commits. Default set is Claude Code only. |
| `setup upgrade` | [`upgrade.md`](upgrade.md) | Refresh the snapshot per `<committed-lock>`, reconcile overrides, refresh symlinks. **Main-checkout only** — worktrees pick it up through the symlink `worktree-init` installs. |
| `setup worktree-init` | [`worktree-init.md`](worktree-init.md) | **Worktree-only.** Symlink the worktree's `<snapshot-dir>` to the main checkout's so both share one framework state. No fetch, no lock files written; idempotent. |
Expand Down
Loading