From bf708230ca86f55d0cd99aa5df8504cd91bac53b Mon Sep 17 00:00:00 2001 From: andy Date: Sun, 25 Oct 2026 00:29:07 -0700 Subject: [PATCH 1/2] feat(web): add skill tree hash spec, shared vectors, and read-only sweep Phase 1, PR 1 of .agents/plans/skill-evals-and-packages.plan.md. - docs/SKILL_TREE_HASH.md: normative agentvouch-tree-v1. It documents what computeTreeHash already does (no stored hash changes), plus path rules, the file manifest, client download checks, and known gaps. - packages/agentvouch-protocol/test-vectors/: 10 vectors and 9 invalid paths, with a stdlib Python reference that produced the expected values. - web/__tests__/lib/treeHashVectors.test.ts: production computeTreeHash matches every vector. - web/lib/treeHashAudit.ts and web/scripts/tree-hash-sweep.ts (npm run db:tree-hash-sweep): read-only audit of stored versions. One SELECT plus Blob reads; it never uses the download routes, which record download events. - AGENTS.md: list the spec as a source of truth. Verified: format:check, lint, typecheck, web vitest (157 files, 1477 tests), CLI vitest (16 files, 119 tests), Python vector check, and a mutation check (locale sort, code point sort, and NFC paths each fail the vectors). next build --webpack compiled and typechecked, then stopped at the /sitemap.xml prerender without DATABASE_URL. Not verified: the sweep has not run on production data. --- .../plans/skill-evals-and-packages.plan.md | 634 ++++++++++++++++++ AGENTS.md | 1 + docs/SKILL_TREE_HASH.md | 204 ++++++ .../test-vectors/tree-hash.json | 328 +++++++++ .../test-vectors/tree_hash_reference.py | 231 +++++++ web/__tests__/lib/treeHashAudit.test.ts | 346 ++++++++++ web/__tests__/lib/treeHashVectors.test.ts | 172 +++++ .../scripts/tree-hash-sweep-source.test.ts | 42 ++ web/lib/treeHashAudit.ts | 476 +++++++++++++ web/package.json | 1 + web/scripts/tree-hash-sweep.ts | 247 +++++++ 11 files changed, 2682 insertions(+) create mode 100644 .agents/plans/skill-evals-and-packages.plan.md create mode 100644 docs/SKILL_TREE_HASH.md create mode 100644 packages/agentvouch-protocol/test-vectors/tree-hash.json create mode 100644 packages/agentvouch-protocol/test-vectors/tree_hash_reference.py create mode 100644 web/__tests__/lib/treeHashAudit.test.ts create mode 100644 web/__tests__/lib/treeHashVectors.test.ts create mode 100644 web/__tests__/scripts/tree-hash-sweep-source.test.ts create mode 100644 web/lib/treeHashAudit.ts create mode 100644 web/scripts/tree-hash-sweep.ts diff --git a/.agents/plans/skill-evals-and-packages.plan.md b/.agents/plans/skill-evals-and-packages.plan.md new file mode 100644 index 00000000..793e371f --- /dev/null +++ b/.agents/plans/skill-evals-and-packages.plan.md @@ -0,0 +1,634 @@ +--- +name: skill-evals-and-packages +overview: "Add version-pinned evidence for skills (static checks, behavior evaluations, safety evaluations) as signed public reports. Use that evidence to publish small, curated, pinned skill packages that install through the AgentVouch CLI and agent plugin marketplaces. This work is off the Base launch path. It changes no on-chain program, economics, custody, or money flow." +todos: + - id: design-approval + content: "Owner reviews decisions D1-D14 and open questions Q1-Q8. Record each answer as a dated note. Done when each decision is Approved, Changed, or Deferred." + status: in_progress + - id: p1-tree-hash-spec + content: "Write docs/SKILL_TREE_HASH.md (the exact tree-hash algorithm, path rules, sort order) with shared test vectors. Done when web computeTreeHash and the new CLI implementation pass the same vectors." + status: in_progress + - id: p1-hash-sweep + content: "Add a read-only script that recomputes tree_hash from served bytes for every public skill version and lists mismatches. Done when production shows zero mismatches, or each mismatch has a recorded fix." + status: in_progress + - id: p1-cli-verify + content: "Make agentvouch skill install and update recompute the tree hash after download and stop on mismatch. Also reject trees where a path is both a file and a folder (at upload in normalizeSkillTreeFiles, and at install). Done when tampered-download tests fail the install, clean installs pass, and the CLI passes the shared tree-hash vectors." + status: pending + - id: p1-version-pinned-reads + content: "Add a read-only, version-pinned archive download (same entitlement rules as today for paid skills). Done when a runner can fetch version N of a free skill and the bytes match the stored tree_hash." + status: pending + - id: p1-static-checks + content: "Add deterministic static checks over the full tree (Agent Skills frontmatter rules, hidden Unicode, remote-install instructions, URL inventory, secret patterns), cached by (tree_hash, checks_version) and shown as a display-only signal. Done when each rule has tests and mirrored skills show results." + status: pending + - id: p1-mirror-provenance + content: "Store upstream commit SHA and path on mirrored and connected-repo skill versions. Add repo-root and frontmatter license fallback with deny-wins rules. Done when new mirror versions show repo@sha:path and license source." + status: pending + - id: p2-eval-runner + content: "Build evals/skill-behavior: fetch a version, verify its hash, wrap it as a plugin, run claude plugin eval with pinned models and a cost cap, normalize to agentvouch.skill-eval-result/v1. Done when a mock run and one real run produce valid results." + status: pending + - id: p2-launch-suites + content: "Write public capability and trigger suites for the launch skills. Add private holdout suites with public hash commitments. Done when each launch skill has 8 or more public cases and every holdout has a commitment entry." + status: pending + - id: p2-sign-ingest + content: "Add the trusted signing job (in-toto statement in DSSE, Ed25519), evaluation tables, ingest route, hash-chained log, and read APIs. Done when an ingested report verifies on the server and with agentvouch eval verify on a clean machine." + status: pending + - id: p2-evidence-display + content: "Show evidence per version on the skill page, in /api/check (evidence block plus display-only signals), and in the CLI. Done when each result shows model, date, runs, suite, and its limitations text." + status: pending + - id: p3-packages-v1 + content: "Write a separate plan, then build: package manifest and lockfile, curation policy, immutable plugin-shaped archives, agentvouch package install and verify, Claude Code marketplace JSON, /packages pages. Allowlisted publishers only." + status: pending + - id: p4-safety-sandbox + content: "Write a separate plan, then build: containerized safety evaluation with canary secrets, egress allowlist and log, model credential broker, mock side-effect tools, split untrusted and trusted CI jobs, and calibration against evals/skill-scan cases." + status: pending + - id: p5-trust-semantics + content: "Ask first: decide if a deterministic safety-eval failure feeds recommended_action avoid, and if /api/check accepts evidence policies. Done when the owner decision is recorded." + status: pending + - id: p6-deferred + content: "Deferred, each needs its own plan: third-party evaluator reports, multi-harness runs (Harbor or NVIDIA SkillEvaluator), Sigstore/Rekor transparency, on-chain anchoring (ERC-8004 Validation Registry on Base Sepolia), paid evaluation requests, curator stake." + status: pending +isProject: true +--- + +# Skill Evaluations and Skill Packages + + + +> **Start with the plain-language guide:** [What we are building, money limits, and launch steps](../../docs/PLAIN_LANGUAGE_GUIDE.md). +> +> Status: draft architecture for owner review, written 2026-10-24 against `main` at `dfc5fff5`. +> This plan does not approve implementation, deployment, trust-rule changes, new dependencies, or model spending. +> No code was changed and no checks were run while writing it. The repository facts below come from reading the code. + +## Summary + +AgentVouch tells an agent two things about a skill today: + +- **Who stands behind it.** Stake, vouches, author bond, and Reports. +- **What an automated reviewer thinks of its text.** The advisory scan returns `review` or `avoid`. + +AgentVouch cannot yet answer three questions: + +1. Does this skill make an agent better at the task it claims? +2. What does the skill do when an agent runs it with real tools? +3. Is the file that I installed the same file that was checked? + +This plan adds that evidence. Each result applies to one exact skill version (its tree hash). Each result is signed and public. **Evidence never grants `allow`.** Only staked trust can grant `allow`. That is the current rule, and this plan keeps it. + +The plan then uses the evidence to make **skill packages**. A package is a small, versioned bundle of related skills that passed a written policy. A person or agent installs a package with one command. The install checks every file hash. AgentVouch can withdraw ("yank") a package version when new evidence shows a problem. + +## Goal + +- Give agents version-pinned, reproducible, signed evidence about what a skill does and whether it helps. +- Make installs verifiable from download to disk. +- Publish curated packages of related, reputable skills through the channels agents already use. +- Keep the work additive, off the Base launch path, and inside the current trust rules until the owner approves a change. + +## Why now (checked 2026-10-24) + +- **Attacks on skill registries are real and current.** A Cloud Security Alliance note (2026-07-07) summarizes ClawHub findings: Koi Security found 341 malicious skills in 2,857 (February 2026), then 824 in more than 10,700. Unit 42 found a skill padded to about 22 MB to pass scanner limits, and skills with no malware at all that steered the agent during the task (affiliate injection, front-running). +- **Most bad skills use prompt injection.** Snyk's ToxicSkills study (2026-02-05) scanned 3,984 skills: 13.4% had a critical issue, 76 were confirmed malicious, and 91% of the malicious ones used prompt injection. +- **Skills can make agents worse.** SkillsBench (arXiv 2602.12670, NeurIPS 2026) ran 84 tasks with and without skills. Curated skills raised pass rates by about 12 to 16 points on average (the paper reports both numbers), but 16 of 84 tasks got worse (worst case −39.3 points). Self-written skills gave no gain. Two or three skills worked best. The agent harness changed results. +- **Standard tools now exist.** `claude plugin eval` (Claude Code v2.1.269 or later) runs each case with and without a plugin and writes a versioned JSON result. NVIDIA publishes SkillEvaluator (Apache-2.0) and signed skills with skill cards. Claude Code marketplaces support commit pins, archive SHA-256 pins, and delisting. +- **No one ties this evidence to stake.** Scanners and signatures show "unchanged" and "no known bad pattern". They do not show "helps" or "behaves safely with tools", and nothing is at risk when they are wrong. The roadmap says AgentVouch should be "the trust layer they query, not a competing storefront" (`docs/ROADMAP.md`, Known risks). This plan is a direct step toward that. + +## Current state (read in the repo, 2026-10-24) + +| Piece | Where | What it gives | Limit found | +| --- | --- | --- | --- | +| Content-addressed versions | `web/lib/db.ts` (`skill_versions.files`, `tree_hash`, `has_executable`); `web/lib/skillStorage.ts` (`computeTreeHash`) | One stable identity per version | No written spec for other clients | +| Advisory AI scan | `web/lib/ai/scan.ts` (rubric `v1`); `skill_scans` keyed by `(tree_hash, rubric_version, model)` | `review` or `avoid` from the text | Reads text only. Input is capped at 512 KB, so large trees are truncated (`truncated=true`) | +| Capability summary | `web/lib/ai/summarize.ts` | One-line summary and capability phrases | Describes. Does not measure | +| Scanner eval harness | `evals/skill-scan/` (25 cases; north star is unsafe recall) | Measures the scanner | Not evidence about a specific skill | +| Trust signals | `web/lib/trustSignals.ts` (`ai_scan`, `vouched`, `author_bonded`, `registered`, `dispute_free`; `fuseActions`) | A checklist and one rollup action; the scan never grants `allow` | No signal about skill behavior | +| Community mirror | `web/lib/mirror/{sources,sync,github}.ts`; sources `anthropics/skills` and `openai/skills`; daily cron 07:00 UTC | Free, attributed mirror listings | Commit SHA is resolved per sync but not stored per version. License file must be inside the skill folder; a repo-root license is ignored | +| CLI install | `packages/agentvouch-cli/src/lib/{install,metadata}.ts` | Writes `.agentvouch.json` with `tree_hash` | Does not recompute or compare the hash after download | +| Downloads | `/api/skills/{id}/raw`, `/archive`, `/zip` | Current version only | No version-pinned read. `buildZipArchive` sorts with `localeCompare`, which depends on ICU data, so zip bytes are not a safe long-term pin | +| Operator auth | `/api/cron/*` with Bearer `CRON_SECRET` | Pattern for operator-only routes | — | + +## Scope + +### In scope + +- A written tree-hash spec, CLI hash verification, and version-pinned reads. +- Deterministic static checks over the full skill tree. +- Capability and trigger evaluations run by AgentVouch in CI, with signed reports and public read APIs. +- Display of evidence on the skill page, in `/api/check`, and in the CLI. Display only. +- Design of safety evaluations and packages (each needs its own plan before build). +- Mirror provenance (commit SHA, path) and better license detection. + +### Out of scope + +- Any change to the Solana program or the Base contracts. +- Economics, floors, splits, vouch rules, curator stake, or paid packages. +- Any real-money flow, including paid evaluation requests. +- Base mainnet (`eip155:8453`) or anything else gated by `docs/MAINNET_READINESS.md`. +- Evaluation of paid skills (needs an author opt-in design). +- Private customer suites and runtime enforcement (`docs/ROADMAP.md`, "Enterprise workflow validation"). They can reuse this runner and report format later under their own plan. +- Changes to `recommended_action` (phase P5, ask first). + +## Architecture + +### Principles + +1. **Evidence, not authority.** Scans and evaluations can lower trust. They never grant `allow`. Staked trust stays the only path to `allow` (`scan.ts` rule, `fuseActions`). +2. **One version, one subject.** Every result names one `tree_hash`. A new version starts with no evaluation evidence. +3. **Reuse open formats.** Agent Skills spec for skills. `claude plugin eval` layout for cases. in-toto Statement in a DSSE envelope for signed reports. Claude Code marketplace JSON for distribution. +4. **Fail closed on integrity.** A hash mismatch stops the install. +5. **Run untrusted content only in a disposable sandbox with no real secrets.** +6. **Public by default.** Public cases, results, and log. Only holdout cases stay private, and their hashes are public. +7. **Minimal and additive.** New tables, routes, CLI commands, and CI jobs. Existing behavior does not change until the owner approves. +8. **Say what a result does not prove.** Every report carries a limitations statement. + +### Component view + +```mermaid +flowchart LR + subgraph Sources + A1[Author publish
POST /api/skills] + A2[Connected repo sync] + A3[Community mirror
daily cron] + end + A1 --> V[(skill_versions
tree_hash)] + A2 --> V + A3 --> V + V --> L0[L0 static checks
web, deterministic] + V --> L1[L1 AI scan
exists: review/avoid] + V --> Q[Evaluation queue
operator API] + Q --> RJ[CI run job
untrusted content] + RJ --> L2[L2 capability + trigger
claude plugin eval] + RJ --> L3[L3 safety sandbox
phase P4] + L2 --> SJ[CI sign job
trusted, DSSE] + L3 --> SJ + SJ --> IN[POST /api/evals/ingest] + IN --> EV[(eval_runs, eval_reports,
eval_log)] + L0 --> TS[Evidence + signals
/api/check, skill page, CLI] + L1 --> TS + EV --> TS + EV --> CU[Package curation
policy + human review] + CU --> PK[(package versions
manifest + lock)] + PK --> D1[agentvouch package install] + PK --> D2[Claude Code marketplace JSON] + PK --> D3[Git mirror, later
Codex, skills CLI] +``` + +### Evidence layers + +| Layer | What it answers | Runs where | Cost | Output | Status | +| --- | --- | --- | --- | --- | --- | +| L0 static checks | Is the tree well formed? Does it hide text or tell the agent to fetch and run remote code? | Web, deterministic, full tree | Near zero | Findings list | New (P1) | +| L1 advisory AI scan | Does the text show concrete risk? | Web, one budgeted model call | Low | `review` or `avoid` | Exists | +| L2 capability and trigger evaluation | Does the skill help? Does it fire only when it should? | CI runner | Medium | WITH, W/OUT, Δ, trigger precision and recall | New (P2) | +| L3 safety evaluation | What does it do with real tools, secrets, and network? | CI sandbox | Medium | Deterministic violations | New (P4) | +| L4 people | Who will put money or reputation behind it? | Vouches, Reports, curator review | — | Stake, disputes, review notes | Vouches and Reports exist; curator review is new (P3) | + +L0 is cheap and sees the whole tree, so it covers the part of a large tree that L1 truncates. + +### Subject identity and integrity + +- **`tree_hash` (existing).** For each file: `path + "\0" + hex(sha256(bytes))`. Sort the entries. Join them with `"\n"`. Take the SHA-256 of the result. The spec must state three details that the code implies: + - Sort order is JavaScript default string order (UTF-16 code units). This is the same as byte order for ASCII paths. The spec must keep this order so existing hashes stay valid. + - Paths are hashed as stored, with no Unicode normalization. The spec should recommend NFC paths for new uploads and say what happens to non-NFC paths. + - The single-file backfill in `db.ts` uses the same formula, so legacy rows are compatible. +- **`archive_sha256` (new, packages).** SHA-256 of the package archive bytes. AgentVouch builds each archive once at publish time and stores the bytes in Blob. It never rebuilds an archive on request. +- **`upstream` (new).** `{ repo, commit_sha, path }` for mirrored and reference-mode content. The runner recomputes the tree hash from upstream at `commit_sha` and compares it with the stored `tree_hash`. +- **Content-addressed lookups.** `/api/check` with local content can return evidence when the content's tree hash matches a known version. + +### Evaluation suites + +**Formats** + +- **Primary: the `claude plugin eval` case layout** (`schema_version: "1.1"`). Each case is a folder with `prompt.md` (frontmatter: `max_turns`, `timeout_seconds`, `allowed_tools`, `tags`, `runs`) and `graders/*.md`. `case.yaml` adds fixtures (`context.scaffold_script`, `context.add_dirs`). Grader types are `regex`, `tool_used`, `tool_order`, `file_exists`, `llm`, and `baseline`. There are no custom-code graders, so deterministic checks use `regex` over produced files or over the trace. +- **Import: skill-creator `evals/evals.json`.** NVIDIA SkillEvaluator also reads this location. A converter turns each `expectations` item into an `llm` grader and adds a `tool_used: Skill` indicator. The UI labels converted suites. +- **Later: Harbor tasks** (`instruction.md`, `environment/Dockerfile`, `tests/test.sh` that writes a reward). Use them for deterministic verifiers and for runs on other harnesses. + +**Kinds** + +- `capability`: Does the agent do better with the skill? Two arms: with the skill and without it. +- `trigger`: Does the skill fire on the right requests and stay quiet on the wrong ones? Cases tagged `trigger-positive` or `trigger-negative`. Negative cases use `tool_used: Skill` with `min: 0`, `max: 0`, `arm: both`. +- `safety`: Phase P4. +- `package-interaction`: Phase P3. Load all members together and check that no member's score drops. + +**Visibility** + +- `public`: in the repo. Anyone can run it with one command. +- `holdout`: in a private repo, grouped by category (for example `solana-dev`, `agent-payments`, `security-review`, `documents`). AgentVouch publishes each suite's SHA-256 in `evals/skill-behavior/holdout-commitments.json` before first use. Rotate each quarter and publish the retired cases, so anyone can check the old commitments. +- `author`: an `evals/` folder inside the skill tree. AgentVouch runs it and labels the result "author cases". These files are part of the tree hash. + +**Case rules** (from the `claude plugin eval` guidance and SkillsBench) + +- Write 8 to 12 cases per skill: about 6 positive, 2 should-not-trigger, and 2 edge cases. +- Give each case one outcome grader and one process grader. In two-arm runs, `tool_used: Skill` is reported but not scored. That is by design. +- Prefer deterministic graders. Use `llm` graders only for short outputs, with concrete PASS and FAIL text. +- Run 3 times per arm (the default). Pin the agent model and the judge model for each suite version. +- Check for leakage: a skill must not contain the answer to a specific case. +- Report Δ and the per-case spread. Δ below zero means the skill made the agent worse. Show that as a warning. + +### Evaluation runner (P2) + +The runner is a stdlib Python orchestrator in `evals/skill-behavior/`, outside the npm workspaces (the same pattern as `evals/skill-scan`). A GitHub Actions workflow runs it on manual dispatch and on a weekly schedule. It never runs on pull requests from forks. + +```mermaid +sequenceDiagram + participant Op as Operator or schedule + participant RJ as CI run job (untrusted content) + participant AV as agentvouch.xyz API + participant SJ as CI sign job (trusted) + Op->>RJ: dispatch(skill UUID, version, suite ids) + RJ->>AV: GET version-pinned archive + RJ->>RJ: recompute tree_hash, stop on mismatch + RJ->>RJ: wrap as plugin, copy suite into wrapper/evals + RJ->>RJ: claude plugin eval (with and without arms) + RJ->>RJ: redact secrets, normalize result + RJ-->>SJ: results artifact + SJ->>SJ: validate schema, build statement, sign DSSE + SJ->>AV: POST /api/evals/ingest + AV->>AV: verify signature, key window, subject, then store and append log +``` + +Steps: + +1. **Plan.** Read the operator queue or the dispatch input: skill UUID, version, suite ids. +2. **Fetch and verify.** Download the version-pinned archive. Recompute the tree hash. Stop if it differs from the stored value. +3. **Wrap.** Build a temporary plugin folder: `.claude-plugin/plugin.json` with `name` set to the skill slug and `version` set to the first 12 characters of the tree hash, plus `skills//` with the exact tree. Copy the suite into `/evals/`. +4. **Run.** `claude plugin eval --trust-plugin --json results.json --model --judge-model --runs 3 --max-cost-usd --no-publish`. Grant no network tools unless a case needs a named fixture host. Install `bubblewrap` and `socat` on the Linux runner so granted shell commands run in the OS sandbox. +5. **Redact.** Remove the model key and other secret patterns from transcripts. Fail the run if a key pattern appears in any output. +6. **Normalize.** Map `aggregate-result.json` (`schemaVersion: 1`) to `agentvouch.skill-eval-result/v1`. +7. **Hand off.** Pass only the results artifact to the sign job. + +P2 runs only first-party skills and skills from allowlisted publishers. In P2 the agent process can read the model key from its own environment. With no network grants the key has no granted way out, but this is not a strong boundary. Untrusted community skills wait for the P4 sandbox. + +### Safety evaluation (P4, separate plan) + +Outline for the P4 plan: + +- **Container per run.** Non-root. A realistic but fake home folder with canary secrets: `~/.env`, `~/.config/solana/id.json` (fake keypair), `~/.aws/credentials`, `~/.ssh/id_ed25519`, a wallet seed file, and a browser cookie stub. New random values and paths for each run. The runner records the values outside the container. +- **Egress.** Deny by default. Allow only the model broker and named fixture hosts. An egress proxy logs every attempt, allowed or blocked. +- **Model credential broker.** Runs outside the container and adds the real key. The container holds only a run-scoped token with a token budget. +- **Mock side-effect tools.** Wallet, payment, x402, email, and deploy actions go to mocks that record the intended effect and enforce limits (amount at or below a cap, recipient on an allowlist). Use MCP mocks with `expect:` guards where the skill uses MCP. Use command shims where it uses a CLI. +- **Watchers.** Writes outside the workspace (shell profiles, crontab, launch agents, `CLAUDE.md`, `AGENTS.md`, `MEMORY.md`, `SOUL.md`), package installs, downloads of executables, and password prompts. +- **Scenarios.** Main-path tasks taken from the public cases, plus adversarial user turns. +- **Deterministic failures.** These, and only these, make `safety_eval` fail: + - a canary value appears in any egress attempt; + - a network attempt goes to a host that the skill does not declare; + - a write goes to a persistence location; + - a mock side effect is above its cap or was not approved; + - the agent downloads and runs a remote binary. + Anything else is `warn` or `pass`. +- **Calibration.** Convert the unsafe cases in `evals/skill-scan/dataset.json` that have runnable behavior into skills. Target: every such case fails. Also run the 11 benign cases. Target: zero failures. Unsafe recall stays the north star. +- **Known limit.** A skill can detect a sandbox and wait. Random paths and values, a realistic home folder, and varied dates make this harder. Reports and disputes stay the backstop. + +### Signed report format + +Normalized result (sketch; the P2 schema file is the source of truth): + +```json +{ + "schema": "agentvouch.skill-eval-result/v1", + "subject": { + "skill_id": "", + "version": 3, + "tree_hash": "", + "upstream": { "repo": "owner/repo", "commit_sha": "<40 hex>", "path": "skills/x" } + }, + "suite": { "id": "agentvouch/solana-dev-public", "version": "1.0.0", "sha256": "", "kind": "capability", "visibility": "public" }, + "harness": { "name": "claude-plugin-eval", "version": "", "agent_model": "", "judge_model": "", "runs_per_arm": 3, "ablation": "with-without" }, + "environment": { "runner": "github-actions", "image_digest": "", "network_grants": [] }, + "results": { "score_with": 0.9, "score_without": 0.6, "delta": 0.3, "cases_passed": 9, "cases_total": 10, "trigger": { "precision": 1.0, "recall": 0.83 } }, + "cost_usd_estimate": 4.1, + "started_at": "", + "finished_at": "", + "raw": { "aggregate_result_sha256": "", "transcripts_uri": "" }, + "limitations": "Measures task results for the listed cases, model, and harness. It does not show that the skill is safe. A new version has no evidence until it is evaluated." +} +``` + +- **Statement.** in-toto Statement v1. `subject` names the skill version. Its digest map carries the tree hash under an AgentVouch key (proposed: `agentvouchTreeV1`), plus `sha256` of the package archive when one exists. `predicateType` is `https://agentvouch.xyz/attestations/skill-eval/v1`. Confirm the digest-key naming against the in-toto digest-set rules in P2. +- **Envelope.** DSSE with payload type `application/vnd.in-toto+json`, signed with Ed25519. `keyid` is the SHA-256 of the public key. Web and CLI can verify this with built-in crypto, so no new dependency is needed. +- **Keys.** Publish the public keys in `web/public/.well-known/agentvouch.json` under `attestation_keys`, each with `valid_from` and `valid_until`. Keep the private key only in a GitHub environment that the sign job alone can use. Rotate with an overlap window. Old reports stay verifiable. +- **Log.** An append-only `eval_log`: `seq`, `report_hash`, `prev_entry_hash`, `entry_hash = sha256(seq ‖ report_hash ‖ prev_entry_hash)`. Public reads at `GET /api/evals/log`. A Sigstore/Rekor public log and on-chain anchoring can come later (P6). + +### Data model (proposed; additive only) + +All DDL goes in a dedicated initializer with `CREATE TABLE IF NOT EXISTS` and nullable `ADD COLUMN IF NOT EXISTS`. Nothing risky runs at request time (`AGENTS.md` §3, mistake 15). + +| Table | Key columns | Notes | +| --- | --- | --- | +| `skill_static_checks` | `tree_hash`, `checks_version`, `findings JSONB`, `checked_at` | PK `(tree_hash, checks_version)`, same pattern as `skill_scans` | +| `eval_suites` | `id`, `slug`, `version`, `kind`, `visibility`, `format`, `sha256`, `source_uri` | Holdout rows store the hash only | +| `eval_runs` | `id`, `tree_hash`, `skill_db_id`, `skill_version`, `suite_id`, `harness`, `harness_version`, `agent_model`, `judge_model`, `runs_per_arm`, `status`, `cost_usd_estimate`, `started_at`, `finished_at` | Status: `completed`, `partial`, `failed` | +| `eval_reports` | `report_hash` (PK), `run_id`, `predicate_type`, `statement JSONB`, `envelope JSONB`, `keyid`, `verified_at` | Ingest is idempotent on `report_hash` | +| `eval_log` | `seq`, `report_hash`, `prev_entry_hash`, `entry_hash`, `created_at` | Append-only | +| `eval_budget_counters` | `bucket`, `period_start`, `used_usd_micros` | Same pattern as `ai_scan_budget_counters` | +| `skill_versions` (columns) | `source_commit_sha VARCHAR(40)`, `source_path TEXT`, `license_tag`, `license_source` | Nullable additions | +| Packages (P3) | `skill_packages`, `skill_package_versions`, `skill_package_members`, `skill_package_reviews` | Defined in the P3 plan | + +### API surface + +Public reads: + +- `GET /api/skills/{id}/evals` — evidence per version: static checks, scan, evaluation summaries, freshness. +- `GET /api/skills/{id}/versions/{version}/archive` — new version-pinned tar, with the same entitlement rules as `/archive`. +- `GET /api/evals/reports/{report_hash}` — the statement, envelope, and links to redacted transcripts. +- `GET /api/evals/log?after=` — the hash-chained log. +- `GET /api/evals/suites` — public suites and holdout commitments. + +Operator routes (Bearer secret, same pattern as `/api/cron/*`): + +- `GET /api/evals/queue` — versions waiting for evaluation. +- `POST /api/evals/ingest` — accepts a signed report. The server verifies the DSSE signature, the key validity window, the schema, and that the subject version exists. Transport auth is defense in depth. The signature is the real check. + +`/api/check` gains an `evidence` object (static checks, evaluations, freshness) and new display-only signals. `recommended_action` does not change before P5. + +### Trust signals (proposal) + +| Signal | Scope | `pass` | `warn` | `fail` | `unknown` | +| --- | --- | --- | --- | --- | --- | +| `static_checks` | skill | No findings above low | Medium findings, such as remote-install steps or hidden Unicode outside code | Not used in v1 | No checks on record | +| `capability_eval` | skill | AgentVouch-run, this exact tree hash, Δ > 0, score at or above threshold | Δ < 0 (makes agents worse) or score below threshold | Not used (this is not adverse security evidence) | No AgentVouch-run result | +| `safety_eval` (P4) | skill | Clean run on this tree hash | Inconclusive run (timeout, partial) | A deterministic violation from the list above | Not run | +| `eval_freshness` | skill | Evidence matches the current version | Latest evidence is for an older version | — | No evidence | + +Rollup rules: + +- P1 to P4: the new signals are display-only. `recommendedActionFromSignals` keeps reading only its current signal ids. +- P5 (ask first): the proposal is that `safety_eval` `fail` counts like a scan `avoid`. No evaluation result can ever produce `allow`. +- Author-submitted results (if allowed later) show as "self-reported" with `unknown` status until AgentVouch reproduces them. + +### Packages (P3; needs its own plan before build) + +**Definition.** A package is an immutable, versioned list of 2 to 4 skill versions, a policy result, and evidence links. SkillsBench found that 2 to 3 skills help most and 4 or more help much less, so v1 caps a package at 4 members. + +**Manifest** (sketch, `agentvouch.package/v1`): + +```json +{ + "schema": "agentvouch.package/v1", + "name": "solana-builder", + "version": "1.0.0", + "title": "Solana Builder", + "curator": "agentvouch", + "members": [ + { + "skill_id": "", + "version": 4, + "tree_hash": "", + "distribution": "hosted", + "license": "MIT", + "evidence": ["sha256:", "sha256:"] + }, + { + "skill_id": "", + "version": 2, + "tree_hash": "", + "distribution": "reference", + "upstream": { "repo": "owner/repo", "commit_sha": "<40 hex>", "path": "skills/x" }, + "license": "CC-BY-SA-4.0", + "evidence": ["sha256:"] + } + ], + "archive": { "sha256": "", "url": "https://agentvouch.xyz/api/packages/solana-builder/1.0.0/plugin.zip" }, + "policy": { "version": "1", "result": "pass", "checked_at": "" } +} +``` + +**Curation policy v1** (proposal; owner approval needed). A member version qualifies only if all of these are true: + +1. **Provenance.** It comes from an allowlisted publisher (GitHub org id match), or from a registered AgentVouch author (`publisher_tier` `registered` or `bonded`). +2. **License.** The license is recorded, and the distribution mode is allowed by that license. +3. **Static checks.** No finding above medium. +4. **AI scan.** Latest rubric result is not `avoid` and not truncated. +5. **Capability evaluation.** AgentVouch ran it on this exact tree hash. Δ ≥ 0 on the public suite and on the category holdout. Score at or above the calibrated threshold. +6. **Safety evaluation.** Required after P4. Required before P4 for any skill that can move funds or send messages (for example a wallet or x402 skill). Those skills wait for P4. +7. **Reports.** The author has no active or upheld Reports (wallet authors). +8. **Human review.** A curator recorded who reviewed it, when, and notes. + +**Interaction check.** Run a package suite with all members loaded. No member's Δ may drop more than a set amount compared with its solo result. + +**Distribution channels** + +| Channel | How a user installs | Verification | Phase | +| --- | --- | --- | --- | +| AgentVouch CLI | `agentvouch package install [@version] --target claude\|codex\|agents [--dir .]` | Full: tree hashes, archive SHA-256, report signatures, yank status, local policy. Writes `agentvouch.lock.json` | P3 first | +| Claude Code marketplace (URL source) | `claude plugin marketplace add https://agentvouch.xyz//marketplace.json`, then `claude plugin install @agentvouch` | Archive source with `sha256` pin for hosted content. `github` or `git-subdir` source with a 40-character `sha` for reference content | P3 | +| Git mirror repo | `codex plugin marketplace add ...` or `npx skills add ...` | Commit pins only. The skills CLI has no lockfile, so label this channel as convenience | Later | + +Notes from the Claude Code docs (checked 2026-10-24): + +- A marketplace added by URL cannot use relative plugin paths. Every entry needs an object source (`archive`, `github`, `git-subdir`). +- One plugin per package fits hosted members in one archive. Reference members cannot be in an AgentVouch archive. The P3 spike must test a package plugin that lists reference members as plugin `dependencies`. +- The archive must be plugin-shaped: `.claude-plugin/plugin.json`, `skills//...` (exact member trees), license texts, and the package manifest. + +**Licensing modes** + +- `hosted`: AgentVouch stores and serves the bytes. Only for licenses on the permissive allowlist (today Apache-2.0, MIT, BSD, ISC). +- `reference`: AgentVouch stores evidence and the pin, not the bytes. Install fetches from upstream at the pinned commit. Use it for every other license (for example Trail of Bits skills are CC-BY-SA-4.0). +- AgentVouch can evaluate content in either mode. Confirm that each source's license allows that use, and publish summaries and short excerpts, not full content. + +**Updates.** The daily mirror finds a new upstream tree hash and creates a new skill version. Static checks and the scan run as today. Package members go to the evaluation queue. If the new version passes the policy, AgentVouch creates a candidate package version. A curator approves it. Users stay pinned until they run `agentvouch package update`. + +**Yank.** These events can yank a package version: an upheld Report against a member author, a safety failure on a pinned version, an upstream compromise notice, a license change, or a curator decision. A yank: + +- marks the version `yanked` with a reason; +- removes it from the marketplace JSON, or points the entry back to the last good version; +- makes `agentvouch package install` refuse it and `agentvouch package verify` warn about it. + +Claude Code can also uninstall delisted plugins on user machines when a marketplace sets `forceRemoveDeletedPlugins: true`. The uninstall happens only when the user's copy of the marketplace refreshes. Third-party marketplaces do not auto-update by default, so this reaches users who turned on auto-update or who update the marketplace by hand. Whether to set it is decision D12. + +### CLI changes + +- P1: `skill install` and `skill update` recompute the tree hash and stop on mismatch. No bypass flag unless the owner asks for one. +- P2: `agentvouch eval show [--version N] [--json]` and `agentvouch eval verify [--tree ]`. `verify` checks the DSSE signature against the published keys and, with `--tree`, the subject hash against local files. +- P3: `agentvouch package list|show|install|update|verify`. `verify` re-hashes installed folders against `agentvouch.lock.json` and reports drift. Drift can mean local edits or a skill that rewrote itself. + +### Web changes + +- **Skill page.** An "Evidence" panel for the shown version: static checks, scan, capability (WITH, W/OUT, Δ, runs, model, date), trigger accuracy, and later safety. Show a freshness warning when evidence is for an older version. Extend the existing `web/components/TrustSignalChecklist.tsx` for the new signals. Put the "why" text in `InfoTip`. Follow `AGENTS.md` §6 (Coral Terminal): stat rows, verdict pills, tokens only, light and dark themes. +- **Packages (P3).** `/packages` and `/packages/[name]`: members, curator, policy result, evidence, and install commands for each channel. + +## Launch content (proposal; owner to confirm) + +First evaluation suites (P2): + +| Skill | Source | Reason | License status | +| --- | --- | --- | --- | +| `agentvouch` | First-party (`web/public/skill.md`, `.cursor/skills/agentvouch/`) | Dogfood the check-before-install flow | First-party | +| `find-skills`, `financial-analysis`, `frontend-design`, `fix-bugbot-pr-comments` | First-party listings (`published-skills/local-skill-classification.json`) | Already listed | First-party | +| Selected Anthropic and OpenAI mirror skills | `anthropics/skills`, `openai/skills` | Reputable, already mirrored | Per skill, through the mirror gate | +| `solana-dev` | `solana-foundation/solana-dev-skill` | Solana builders; matches AgentVouch users | To verify (not stated on solana.com/skills) | +| `agentic-wallet` | `coinbase/agentic-wallet-skills` | Agent payments and x402 | MIT per the repo README. Moves funds, so it waits for P4 | +| Selected Trail of Bits audit skills | `trailofbits/skills` | Security review | CC-BY-SA-4.0, so reference mode only | + +Candidate packages (P3): + +1. **AgentVouch Trust Kit:** `agentvouch` and `find-skills` — check trust before install. +2. **Solana Builder:** `solana-dev` and the AgentVouch trust check. +3. **Agent Payments:** `agentic-wallet` and the AgentVouch trust check. Blocked until P4. +4. **Security Review:** a Trail of Bits subset in reference mode, after license review. +5. **Document Tools:** Anthropic document skills only if their license allows the chosen mode. + +## Decisions (proposed; each needs owner approval) + +| ID | Decision | Recommendation | Main alternative | Status | +| --- | --- | --- | --- | --- | +| D1 | Can evidence grant `allow`? | No. Evidence can only lower trust or inform | Let strong evidence raise trust | Proposed (keeps the current rule) | +| D2 | Version identity | Existing `tree_hash` with a written spec, plus archive SHA-256 and upstream commit SHA | A new hash scheme | Proposed | +| D3 | Case format | `claude plugin eval` layout; import `evals/evals.json`; Harbor later | A custom AgentVouch format | Proposed | +| D4 | Where runs happen | GitHub Actions; stdlib Python orchestrator outside npm workspaces; separate run and sign jobs; never on fork PRs | Vercel Sandbox started from the web app (good later for on-demand runs) | Proposed | +| D5 | What P2 may run | First-party and allowlisted publishers only; community skills wait for P4 | Run everything from day one | Proposed | +| D6 | Report signing | in-toto Statement in DSSE, Ed25519 key in a GitHub environment, public keys in `.well-known`, hash-chained log | Sigstore keyless now (no key custody, but a new verification dependency) | Proposed | +| D7 | Holdouts | Private repo, public hash commitments, quarterly rotation, reveal retired cases | Public cases only | Proposed | +| D8 | Trust rollup | New signals are display-only until P5 | Feed `recommended_action` now | Proposed | +| D9 | Curation | AgentVouch is the only curator in v1; 2 to 4 skills per package; pinned; policy plus human review; no payments, no stake | Open curation | Proposed | +| D10 | Distribution order | CLI first, then the Claude Code marketplace URL, then a Git mirror | Git mirror first (reaches Codex and the skills CLI sooner, weaker pins) | Proposed | +| D11 | Licensing | Host only permissive licenses (existing gate plus repo-root and frontmatter fallback, deny wins); reference mode for all others | Host anything with attribution | Proposed | +| D12 | Revocation reach | Yank and delist; leave `forceRemoveDeletedPlugins` off until the owner decides | Turn it on at launch | Open (Q4) | +| D13 | Spend | Hard caps per run (`--max-cost-usd`) and per month (`eval_budget_counters`) | No cap | Open (Q5) | +| D14 | Chain | No on-chain change. On-chain anchoring later; the ERC-8004 Validation Registry on Base Sepolia is the main candidate | Anchor now | Proposed | + +## Phases + +Each phase is one or more small PRs off current `main`. P3 and later need their own plan files before build. + +**P0 — Design lock (this plan).** The owner answers Q1 to Q8 and approves or changes D1 to D14. Then write `.agents/plans/skill-packages-v1.plan.md` and `.agents/plans/skill-safety-sandbox.plan.md`. + +**P1 — Integrity and static evidence.** Small and useful alone. It also unblocks every later phase. + +- `docs/SKILL_TREE_HASH.md` (new): algorithm, path rules, sort order, test vectors. +- Shared vectors file (proposed: `packages/agentvouch-protocol/test-vectors/tree-hash.json`), tested by web and CLI. +- `web/scripts/tree-hash-sweep.ts` (new, read-only): recompute and compare every public version. Run it before CLI verification ships. +- `packages/agentvouch-cli/src/lib/integrity.ts` (new); changes in `install.ts`, `update.ts`, `archive.ts`; tests with tampered archives. +- `web/app/api/skills/[id]/versions/[version]/archive/route.ts` (new). +- `web/lib/skillStaticChecks.ts` (new) and its table; a display-only `static_checks` signal in `web/lib/trustSignals.ts`. +- `web/lib/mirror/sync.ts` and `web/lib/mirror/github.ts`: store commit SHA, path, and license source; add the license fallback. +- `web/public/skill.md`: document the version-pinned read and CLI verification. + +**P2 — Capability and trigger evaluations.** + +- `evals/skill-behavior/` (new): `README.md`, `run.py`, `normalize.py`, `schemas/skill-eval-result.v1.json`, `suites//`, `holdout-commitments.json`, and mock fixtures. +- `.github/workflows/skill-evals.yml` (new): manual dispatch and a weekly schedule. Job `run` (environment `eval-run`, secret: budget-capped model key). Job `sign-ingest` (environment `eval-signing`, secrets: signing key, ingest secret). +- `web/lib/evals/` (new): `types.ts`, `dsse.ts`, `verify.ts`, `store.ts`, `queries.ts`; an additive initializer for the tables. +- Routes (new): `web/app/api/evals/{ingest,queue,log,suites}/route.ts`, `web/app/api/evals/reports/[hash]/route.ts`, `web/app/api/skills/[id]/evals/route.ts`. +- `web/app/api/check/route.ts` (evidence block); `web/lib/trustSignals.ts` (`capability_eval`, `eval_freshness`, display-only). +- `web/app/skills/[id]/SkillDetailClient.tsx`, `web/components/TrustSignalChecklist.tsx`, and a new `web/components/EvidencePanel.tsx`. +- CLI: `eval show`, `eval verify`. +- `web/public/.well-known/agentvouch.json` (`attestation_keys`) and `web/public/skill.md`. + +**P3 — Packages v1** (separate plan). Manifest and lockfile spec, tables, policy engine, immutable archive builder (byte-order sort, stored in Blob), package routes, marketplace JSON route, `/packages` pages, CLI package commands, and the launch packages from allowlisted publishers. + +**P4 — Safety sandbox** (separate plan). Container image with pinned digest, canaries, egress proxy and log, model broker, mock side-effect tools, watchers, split jobs, calibration report, and the `safety_eval` signal (display-only). + +**P5 — Trust semantics** (ask first). Safety failure in the rollup; evidence policies in `/api/check` (for example `require: ["safety_eval:pass"]`); an MCP read tool for evidence when the MCP surface exists. + +**P6 — Deferred.** Third-party evaluator reports; runs on other harnesses (Harbor, NVIDIA SkillEvaluator); Sigstore/Rekor; on-chain anchoring; paid evaluation requests (a money flow, so it inherits launch gates); curator stake (an economics change). + +## Verification + +- **Every web or CLI phase:** the full local gate from `AGENTS.md` §2, with Node 24 selected as `AGENTS.md` §1 says (`. "$HOME/.nvm/nvm.sh" --no-use && { nvm use --silent || nvm install; }`): `npm run format:check`, `npm run lint`, `npm run typecheck`, `npm test`, and `npm exec --workspace @agentvouch/web -- next build --webpack`. +- **P1:** + - Tree-hash vectors pass in web and CLI tests. + - The sweep reports zero mismatches on production before fail-closed CLI verification ships. + - CLI tests: a changed byte, an added file, and a renamed file each fail the install; a clean install passes. + - Static-check tests cover each rule, including zero-width and bidi characters, `curl ... | sh`, password-protected archives, and secret patterns. +- **P2:** + - `python3 evals/skill-behavior/run.py --mock` validates normalization against fixture results. + - One real run per launch skill. + - Reproducibility: run each launch suite twice; Δ must stay inside a tolerance (start at ±0.15, then calibrate). + - `agentvouch eval verify --tree ` passes on a clean machine with no repo checkout. + - Ingest rejects a bad signature, an expired key, an unknown subject, and a schema error. + - `/api/check` output keeps the same `recommended_action` for a fixture set before and after the change. +- **P3:** `claude plugin validate` on the generated marketplace; install from the URL marketplace on a clean machine; archive SHA-256 matches the manifest; yanked versions are refused by the CLI and missing from the marketplace JSON. +- **P4:** the calibration report (per case) and zero failures on the benign set. + +## Rollout and rollback + +- Flags: + - `AGENTVOUCH_EVALS_INGEST_ENABLED` (server): when off, the ingest route refuses requests. + - `NEXT_PUBLIC_AGENTVOUCH_EVIDENCE_UI_ENABLED` (render): the evidence panel. + - `AGENTVOUCH_PACKAGES_ENABLED` (server) and `NEXT_PUBLIC_AGENTVOUCH_PACKAGES_UI_ENABLED` (render). + - Render-affecting defaults read only `NEXT_PUBLIC_*` values (`AGENTS.md` §3, mistake 5). +- Rollback: turn the flags off. Tables are additive and stay in place. The marketplace route returns an empty plugin list. A false hash mismatch is fixed in server data, not by weakening the CLI. +- Public copy says "devnet beta" and labels unfinished parts `WIP`, as the docs rules require. + +## Risks and mitigations + +| Risk | Mitigation | +| --- | --- | +| People read "evaluated" as "safe" | Limitations text on every report; evidence never grants `allow`; copy review | +| Overfitting to public cases | Holdouts with public commitments; author cases labeled | +| Sandbox evasion (skill waits or detects the eval) | Random canaries and paths, realistic home folder; Reports and disputes stay the backstop | +| Judge model is prompt-injected by skill output | Deterministic graders for safety; strict PASS/FAIL rubrics; spot-check about 20 judge grades per suite | +| Model key leaks into transcripts | Dedicated low-limit key, redaction, run fails on key patterns, rotate on any hit; broker in P4 | +| Cost overrun | Per-run and monthly caps; pinned cheaper model for wide sweeps | +| License violation | Host only permissive content; reference mode; license and attribution files in archives | +| Auto-uninstall surprises users | Owner decision (Q4); clear docs before turning it on | +| Results change when models change | Pin models; show the model on every result; re-baseline each quarter | +| Claude-only harness bias | Label the harness; other harnesses in P6 (SkillsBench shows harness effects) | +| Work competes with launch gates | Off the critical path; small PRs; P3 and later need their own plans | + +## Cost model (estimate; measure in P2) + +- One suite of 10 cases × 3 runs × 2 arms is 60 agent runs, plus up to 3 judge calls per `llm` grader per run. +- If one agent run costs $0.05 to $0.40 (model, turns, and context decide this), one suite run costs about $3 to $24 plus judge calls. `claude plugin eval` prints a list-price estimate, so P2 replaces this guess with real numbers. +- Launch estimate: about 20 skill versions × 2 suites (public and holdout) is 40 suite runs, about $120 to $960. New versions and quarterly re-baselines add to this. +- GitHub-hosted runner minutes are free for public repositories. Confirm repo visibility. + +## Open questions + +- **Q1 — Order.** Recommended: P1, then P2, then P3, then P4. Alternatives: packages first, or the safety sandbox first. +- **Q2 — Who may submit results in v1?** Recommended: AgentVouch only. Alternatives: authors (labeled self-reported) or third-party evaluators. +- **Q3 — Launch list.** Confirm or change the launch skills and packages above. +- **Q4 — Auto-uninstall.** Should the AgentVouch marketplace set `forceRemoveDeletedPlugins: true` so a yank uninstalls the plugin for users with auto-update on? +- **Q5 — Monthly evaluation spend cap** during beta. +- **Q6 — Repos.** Recommended: public cases and runner in this repo, holdouts in a private repo. Confirm the private repo name and who can access it. +- **Q7 — Signing.** Recommended: Ed25519 now, Sigstore later. Alternative: Sigstore keyless now. +- **Q8 — Visibility.** Show evidence and packages on agentvouch.xyz before mainnet (labeled devnet beta)? + +## Blockers and assumptions + +- Assumes `dirtybits/agentvouch` stays public (`web/public/skill.md` links it publicly). Not checked from this session. +- `claude plugin eval` requires Claude Code v2.1.269 or later, git 2.31 or later, and an API key in CI. Its output format is `schemaVersion: 1`. The P2 runner must pin a Claude Code version. +- The runner needs the version-pinned read from P1. +- New dependencies (for example a Sigstore library or `@vercel/sandbox`) need owner approval first (`AGENTS.md` §5). +- Licenses for `solana-foundation/solana-dev-skill`, the NVIDIA skills, and the Anthropic document skills are not yet checked. +- Paid-skill evaluation is out of scope until an author opt-in design exists. + +## How this connects to existing strategy + +- **Skill value thesis** (`docs/ROADMAP.md` Phase D): curation should favor domain-specific, fresh, executable, safe skills. Evidence makes that measurable. +- **"Be the trust layer they query":** packages install through native channels (Claude Code and Codex marketplaces, skills CLI) and carry AgentVouch evidence with them. +- **Enterprise workflow validation** (ROADMAP discovery track): it can reuse the runner, result schema, and signed reports with private suites under a separate plan. +- **Evidence hashing for disputes** (ROADMAP Phase C): a signed report hash is ready-made evidence for a Report. +- **MCP adoption surface:** a later read-only tool can return the same evidence object as `/api/check`. + +## Sources (checked 2026-10-24) + +- Claude Code — Test plugins with evals: https://code.claude.com/docs/en/plugin-evals +- Claude Code — Plugin loading reference: https://code.claude.com/docs/en/plugins/loading.md +- Claude Code — Marketplace reference (Japanese page; same fields): https://code.claude.com/docs/ja/plugins/marketplace-reference.md +- Claude Code — Create a marketplace: https://code.claude.com/docs/en/plugin-marketplaces +- SkillsBench (arXiv 2602.12670): https://arxiv.org/html/2602.12670v1 +- CSA research note on ClawHub supply-chain risk (2026-07-07): https://labs.cloudsecurityalliance.org/research/csa-research-note-openclaw-clawhub-supply-chain-risk-2026070/ +- Snyk ToxicSkills: https://snyk.io/blog/toxicskills-malicious-ai-agent-skills-clawhub +- NVIDIA — Evaluate agent skills before publication: https://docs.nvidia.com/skills/evaluating-agent-skills +- NVIDIA — Signing agent skills: https://docs.nvidia.com/skills/signing-agent-skills.md +- NVIDIA — Verified agent skills: https://developer.nvidia.com/blog/nvidia-verified-agent-skills-provide-capability-governance-for-ai-agents/ +- NVIDIA SkillEvaluator: https://github.com/NVIDIA/SkillEvaluator +- Agent Skills specification (mirror of the spec text): https://moltchat-agent-commons.onrender.com/wiki/Agent_Skills_specification_(SKILL.md_format%2C_verbatim) +- skill-creator eval schemas (published copy): https://cdn.jsdelivr.net/npm/opencode-skill-creator@0.2.25/dist/skill/references/schemas.md +- SkillsBench Harbor task layout: https://skills.cat/skills/benchflow-ai/skillsbench/harbor +- skills.sh FAQ: https://skills.sh/docs/faq +- Vercel Sandbox egress firewall: https://vercel.com/changelog/advanced-egress-firewall-filtering-for-vercel-sandbox +- ERC-8004 contract deployments: https://erc-8004.quicknode.com/docs/contracts +- Trail of Bits skills: https://github.com/trailofbits/skills +- Coinbase agentic wallet skills: https://cdn.jsdelivr.net/gh/coinbase/agentic-wallet-skills@main/README.md +- Solana skills: https://solana.com/skills + +## Dated notes + +- 2026-10-24: Plan drafted after reading the repo at `main` `dfc5fff5` and the sources above. Found in code: the CLI stores but does not verify `tree_hash`; there is no version-pinned read; the mirror resolves a commit SHA per sync but does not store it per version; zip builds sort with `localeCompare`. No code changed. No checks run. +- 2026-10-24: The owner approved starting Phase 1 in chat. This answers Q1 for now (P1 first). D1-D14 and Q2-Q8 are still open, so `design-approval` stays in progress. +- 2026-10-24 (`p1-tree-hash-spec`, branch `feat/skill-tree-hash-spec`): Added `docs/SKILL_TREE_HASH.md`, 10 vectors plus 9 invalid paths in `packages/agentvouch-protocol/test-vectors/tree-hash.json`, the stdlib Python reference `tree_hash_reference.py` (it produced the expected values), and `web/__tests__/lib/treeHashVectors.test.ts`. Production `computeTreeHash` and the Python reference agree on every vector. Mutation check: locale-aware sorting, code point sorting, and NFC path normalization each fail the vectors. The CLI half of Done-when lands with `p1-cli-verify`, so this todo stays in progress. +- 2026-10-24 (`p1-hash-sweep`): Added `web/lib/treeHashAudit.ts` (pure audit, the SELECT-only query builder, behavioral tests) and `web/scripts/tree-hash-sweep.ts` (`npm run db:tree-hash-sweep`). The sweep reads the database and Blob storage directly. It does not use the public download routes, because they record download events. It also checks that `skill_versions.content`, which `/raw` serves for `SKILL.md`, equals `SKILL.md` in the stored tree. It has not run on production yet: it needs `DATABASE_URL` and `BLOB_READ_WRITE_TOKEN` from the owner's environment. +- 2026-10-24: New finding: upload does not reject a tree where one path is also a folder of another path (`docs` and `docs/a.md`). Such a tree cannot be written to disk. The sweep reports it as `path_conflict`. The fix moved into `p1-cli-verify`. +- 2026-10-24: New finding (not yet seen in data): `ingestTarArchive` enforces the upload limits (200 files, 5 MB), but mirror sync stores trees up to 512 files and 8 MB. `/raw?path=` and `/zip` read stored trees through it, so they would fail for a mirrored tree above the upload limits; `/archive` would not. The sweep reports such versions as `files_unavailable` with the limit message. If the production sweep finds any, fix the reader to accept the limits the tree was stored with. diff --git a/AGENTS.md b/AGENTS.md index 8ddccc13..14d9cc16 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,6 +19,7 @@ This file is the operating manual for any model working in this repo. `CLAUDE.md - Cross-chain instruction/capability parity and source-vs-deployment status: `docs/CHAIN_CAPABILITY_MAP.md`. - Chain seam interfaces: `web/lib/adapters/types.ts`. - Agent-facing product contract: `web/public/skill.md` (canonical served file; keep in sync after protocol/product changes; do not create a duplicate root `SKILL.md`). + - Skill version identity (tree hash `agentvouch-tree-v1`), the file manifest, and client download checks: `docs/SKILL_TREE_HASH.md`. Its vectors (`packages/agentvouch-protocol/test-vectors/tree-hash.json`) must keep passing in web, CLI, and the Python reference; never change v1 in place. ## 1. Session start checklist diff --git a/docs/SKILL_TREE_HASH.md b/docs/SKILL_TREE_HASH.md new file mode 100644 index 00000000..a11a2045 --- /dev/null +++ b/docs/SKILL_TREE_HASH.md @@ -0,0 +1,204 @@ +# Skill Tree Hash (`agentvouch-tree-v1`) + + + +> Status: normative, version 1. Written 2026-10-24. +> It describes what `computeTreeHash` in `web/lib/skillStorage.ts` already does at `main` `dfc5fff5`. It does not change any stored hash. +> Test vectors: `packages/agentvouch-protocol/test-vectors/tree-hash.json`. +> Plan: `.agents/plans/skill-evals-and-packages.plan.md` (Phase 1). + +## What the tree hash is + +The tree hash is the fingerprint of one skill version. It covers the path and the exact bytes of every file in the version. If one byte or one file name changes, the tree hash changes. + +AgentVouch uses the tree hash to: + +- name one exact version (scan results in `skill_scans` use it as their key); +- let a client check that downloaded files are the files that AgentVouch stored and reports. + +A matching tree hash proves that the bytes are unchanged. It does not prove that the content is safe. Trust evidence (stake, Reports, scans, evaluations) is separate. + +## Terms + +- **Skill tree:** the files of one skill version. It contains exactly one top-level `SKILL.md`. +- **Path:** the location of a file in the tree, relative to the tree root, with `/` between segments. +- **Digest:** the lowercase hexadecimal SHA-256 of the exact bytes of one file (64 characters). + +## Path rules + +A stored path obeys all of these rules. Upload enforces them (`normalizeSkillPath` and `normalizeSkillTreeFiles` in `web/lib/skillStorage.ts`). + +1. It is not empty. +2. It has no white space at the start or end. Upload removes it. +3. It is relative. It does not start with `/` or with a drive letter such as `C:/`. +4. It uses `/` only. Upload changes `\` to `/`, so a stored path never contains `\`. +5. It has no empty segment (`a//b`, `dir/`), no `.` segment, and no `..` segment. +6. It is unique in the tree. +7. One path is exactly `SKILL.md`. + +AgentVouch stores and hashes path text exactly as it was received, as UTF-8. It applies no Unicode normalization. Two names that look the same can be different paths: `café.md` written with `é` (U+00E9, NFC) and `café.md` written with `e` plus a combining accent (U+0065 U+0301, NFD). They give different tree hashes. Publishers should use NFC names. + +**Client rule:** a client must also reject a tree in which one path is a folder of another path, for example `docs` and `docs/a.md`. Such a tree cannot be written to a file system. Upload does not enforce this yet (see [Known gaps](#known-gaps-2026-10-24)). + +## Algorithm + +Input: the path and the bytes of every file in the tree. + +1. For each file, compute its digest: the lowercase hex SHA-256 of its exact bytes. Do not change line endings, text encoding, or a byte order mark. +2. For each file, make one entry: the path, then one U+0000 (NUL) character, then the digest. +3. Sort the entries in ascending UTF-16 code unit order. This is the default JavaScript string sort (`Array.prototype.sort()` with no compare function). +4. Join the sorted entries with one U+000A (LF) between entries. Do not add an LF at the end. +5. Encode the joined text as UTF-8. +6. The tree hash is the lowercase hex SHA-256 of those bytes. + +Notes: + +- **Only paths and bytes go into the hash.** Content type, the executable flag, file mode, timestamps, and the archive format do not. AgentVouch computes the executable flag from the path and the first bytes, so the flag cannot change without a path or byte change. +- **Do not use locale-aware sorting.** For ASCII paths, UTF-16 code unit order is the same as byte order, so uppercase letters sort before lowercase letters: `SKILL.md`, then `Zebra.md`, then `references/api.md`. `localeCompare` and ICU collation give a different order and a wrong hash. +- **Code unit order is not code point order.** For most non-ASCII paths the two orders agree. They differ when a character above U+FFFF meets a character from U+E000 to U+FFFF. Example: `😀` (U+1F600, code units D83D DE00) sorts before `～` (U+FF5E) in code unit order, but after it in code point order. Use code unit order. In Python, sort with `key=lambda entry: entry.encode("utf-16-be")`. +- **Entries are sorted, not paths.** NUL is lower than every other character, so the result is the same as sorting the paths. + +### Reference implementations + +TypeScript (the production code in `web/lib/skillStorage.ts`, reduced): + +```ts +import { createHash } from "node:crypto"; + +export function treeHash(files: { path: string; bytes: Uint8Array }[]): string { + const entries = files + .map((file) => { + const digest = createHash("sha256").update(file.bytes).digest("hex"); + return `${file.path}\0${digest}`; + }) + .sort(); + return createHash("sha256").update(entries.join("\n")).digest("hex"); +} +``` + +Python (standard library; the full version is `packages/agentvouch-protocol/test-vectors/tree_hash_reference.py`): + +```python +import hashlib + +def tree_hash(files): # files: list of (path: str, data: bytes) + entries = ["%s\0%s" % (path, hashlib.sha256(data).hexdigest()) for path, data in files] + entries.sort(key=lambda entry: entry.encode("utf-16-be")) + return hashlib.sha256("\n".join(entries).encode("utf-8")).hexdigest() +``` + +## The manifest + +Each version stores a manifest in `skill_versions.files`. The API returns it as `files` in `GET /api/skills/{id}`. It has one entry per file: + +| Field | Meaning | In the hash | +| ------------- | -------------------------------------------------------- | ---------------- | +| `path` | The stored path | Yes | +| `sha256` | The file digest | Yes (as step 1) | +| `size` | The file length in bytes | No | +| `contentType` | The served content type | No | +| `executable` | Computed from the path and the first bytes | No | + +The manifest digests are the output of step 1, so the tree hash can be computed from the manifest alone. This lets a client check one file without the whole tree. + +## How a client checks a download + +1. Read `tree_hash` and `files` from `GET /api/skills/{id}`. Treat them as the claim to check. +2. **Tree download** (`/api/skills/{id}/archive`): + - Read the tar archive. Reject any entry that is not a regular file. + - Reject any path that breaks the path rules, including a path that is also a folder of another path. + - Compute the tree hash from the extracted bytes. + - Stop if it differs from `tree_hash`. Do not move files to their final place before the check passes. +3. **One file** (`/api/skills/{id}/raw?path=...`): + - Compute the digest of the downloaded bytes. Compare it with the manifest entry for that path. + - Compute the tree hash from the manifest digests. Compare it with `tree_hash`. + - Stop if either check fails. +4. Record the checked tree hash with the install. The CLI writes it to `.agentvouch.json`. + +Note: `/raw` serves `SKILL.md` from `skill_versions.content` for every storage type. The stored tree must therefore hold the same `SKILL.md` bytes. The sweep below checks this. + +## Legacy single-file versions + +Single-file versions that existed before multi-file storage got a manifest and a tree hash from a SQL backfill in `web/lib/db.ts`. That SQL computes the SHA-256 of `SKILL.md`, then NUL, then the hex digest of the content. This is the same as the algorithm for a tree with one file. The vector test checks this equivalence. + +## Test vectors + +`packages/agentvouch-protocol/test-vectors/tree-hash.json` lists inputs and expected results: + +| Vector | What it pins | +| ------------------------------- | ------------------------------------------------------------------- | +| `single-skill-md` | The basic one-file case and the legacy SQL formula | +| `crlf-line-endings` | Bytes are hashed exactly: CRLF and LF differ | +| `empty-file` | A zero-byte file | +| `binary-file` | Raw bytes that are not valid UTF-8 | +| `ascii-case-order` | Uppercase sorts before lowercase; locale-aware sorting fails | +| `punctuation-and-space-order` | Order of space, `-`, `.`, `/`, `_`; a file before a same-name folder | +| `unicode-nfc-path` | A non-ASCII path in NFC | +| `unicode-nfd-path` | The same visible path in NFD gives a different hash | +| `astral-vs-bmp-order` | UTF-16 code unit order, not code point order | +| `nested-tree` | Several nested folders | +| `invalid_paths` (list) | Paths that every implementation must reject | + +Two independent implementations check the vectors: + +- `web/__tests__/lib/treeHashVectors.test.ts` runs the production code. +- `tree_hash_reference.py` is a standard-library Python version. It produced the expected values. Run it with `python3 packages/agentvouch-protocol/test-vectors/tree_hash_reference.py`. + +On 2026-10-24, three wrong implementations were tried against the vectors. Each one failed: locale-aware sorting, code point sorting, and NFC normalization of paths. + +To add a vector: add its inputs, run `tree_hash_reference.py --write`, review the diff, run `npx prettier --write` on the JSON file, then run both checks. + +`tree_hash_reference.py --dir PATH` prints the tree hash of a local folder. It leaves out the same files as `agentvouch` CLI publish: `.git` and `node_modules` folders, `.DS_Store` files, `*.agentvouch.json` files, and symbolic links. + +## Checking stored data + +`web/scripts/tree-hash-sweep.ts` recomputes the tree hash of stored versions and lists every version where the data does not agree. Run it before the CLI starts to stop installs on a hash mismatch. + +It is read-only: + +- It runs one SELECT query (built and tested in `web/lib/treeHashAudit.ts`). It never calls `initializeDatabase()`, which runs schema changes. +- It reads Blob storage. It never writes or deletes. +- It does not call the download routes. Those routes record download events and change public install counts. + +Usage, from `web/`: + +```bash +npx tsx --env-file=.env.local ./scripts/tree-hash-sweep.ts --latest-only +npx tsx --env-file=.env.local ./scripts/tree-hash-sweep.ts --json tree-hash-sweep.json +# or, with the variables exported: +npm run db:tree-hash-sweep --workspace @agentvouch/web -- --skill --verbose +``` + +It needs `DATABASE_URL`, and `BLOB_READ_WRITE_TOKEN` for multi-file versions. It prints the database host before the query. If `EXPECTED_DATABASE_HOST` is set and does not match, it stops before it reads anything. Exit codes: `0` every version matches, `1` one or more versions have findings, `2` setup or usage error. + +Finding codes: + +| Code | Meaning | +| ----------------------------- | -------------------------------------------------------------------------- | +| `missing_tree_hash` | The version has no stored tree hash | +| `malformed_tree_hash` | The stored tree hash is not 64 lowercase hex characters | +| `missing_manifest` | The version has no stored manifest | +| `invalid_manifest` | The manifest is not an array of valid entries | +| `manifest_tree_hash_mismatch` | The manifest digests give a different tree hash | +| `files_unavailable` | The stored bytes could not be read: for example a missing Blob archive, or a tree above the upload limits (see Known gaps) | +| `bytes_tree_hash_mismatch` | The stored bytes give a different tree hash | +| `file_digest_mismatch` | A file's bytes do not match its manifest digest | +| `file_size_mismatch` | A file's length does not match its manifest size | +| `file_missing_from_manifest` | Storage has a file that the manifest does not list | +| `file_missing_from_storage` | The manifest lists a file that storage does not have | +| `raw_skill_md_mismatch` | `skill_versions.content` (served by `/raw`) differs from the tree's `SKILL.md` | +| `invalid_path` | A stored path breaks the path rules | +| `duplicate_path` | A path appears more than once | +| `missing_skill_md` | The tree has no top-level `SKILL.md` | +| `path_conflict` | A path is a file and also a folder of another path | + +## Changes to this algorithm + +A change to any step is a new algorithm with a new name, for example `agentvouch-tree-v2`. Stored hashes keep the version that made them. Never change `agentvouch-tree-v1` in place. + +## Known gaps (2026-10-24) + +- Upload does not reject a tree where one path is also a folder of another path. The sweep reports such trees as `path_conflict`. The fix is planned with the CLI download check (Phase 1, second PR). +- The CLI records `tree_hash` but does not check downloads against it yet (Phase 1, second PR). +- The `/zip` builder sorts entries with `localeCompare`, which depends on the server's ICU data. Zip bytes can differ between servers, so do not pin zip bytes. The tree hash does not depend on archive bytes. +- The stored-tar reader (`ingestTarArchive`) enforces the upload limits (200 files, 5 MB), but mirror sync stores trees up to 512 files and 8 MB. `/raw?path=` and `/zip` use that reader, so they cannot serve a mirrored tree above the upload limits. The sweep reports such a version as `files_unavailable` with the limit message. `/archive` is not affected, because it serves the stored tar bytes directly. diff --git a/packages/agentvouch-protocol/test-vectors/tree-hash.json b/packages/agentvouch-protocol/test-vectors/tree-hash.json new file mode 100644 index 00000000..3882795c --- /dev/null +++ b/packages/agentvouch-protocol/test-vectors/tree-hash.json @@ -0,0 +1,328 @@ +{ + "schema": "agentvouch.tree-hash-vectors/v1", + "algorithm": "agentvouch-tree-v1", + "spec": "docs/SKILL_TREE_HASH.md", + "notes": "Each file is content_utf8 (text) or content_base64 (raw bytes). Non-ASCII text uses \\u escapes on purpose. To change a vector, edit its inputs, run tree_hash_reference.py --write, review the diff, then run prettier on this file.", + "vectors": [ + { + "id": "single-skill-md", + "description": "One SKILL.md file. This is also the shape of legacy single-file versions (see the SQL backfill note in the spec).", + "files": [ + { + "path": "SKILL.md", + "content_utf8": "---\nname: hello-world\ndescription: Say hello. Use when the user asks for a greeting.\n---\n\n# Hello World\n\nReply with one short greeting.\n" + } + ], + "expected": { + "sorted_paths": ["SKILL.md"], + "file_sha256": { + "SKILL.md": "7cfd32d4977363dda1f02aee644675610f1e928f3ab60647784f2f4c3fdc5ac8" + }, + "tree_hash": "d8fbcabf71d10a068ea9907c99c1d0741bcacdb1c71140b53be06b6b511c0fd7" + } + }, + { + "id": "crlf-line-endings", + "description": "Same text as single-skill-md with CRLF line endings. Bytes are hashed exactly, so the digest and the tree hash differ.", + "files": [ + { + "path": "SKILL.md", + "content_utf8": "---\r\nname: hello-world\r\ndescription: Say hello. Use when the user asks for a greeting.\r\n---\r\n\r\n# Hello World\r\n\r\nReply with one short greeting.\r\n" + } + ], + "expected": { + "sorted_paths": ["SKILL.md"], + "file_sha256": { + "SKILL.md": "2b1bb32469d392e90645c18f4db6028c0558d44582c4f3ba32403a8be28ac6e3" + }, + "tree_hash": "463cbad467f7cece5959369b3071cdb99503edaa4528dd77100e7734aaeb5ac1" + } + }, + { + "id": "empty-file", + "description": "A zero-byte file. Its digest is the SHA-256 of empty input.", + "files": [ + { + "path": "SKILL.md", + "content_utf8": "---\nname: hello-world\ndescription: Say hello. Use when the user asks for a greeting.\n---\n\n# Hello World\n\nReply with one short greeting.\n" + }, + { + "path": "assets/empty.txt", + "content_utf8": "" + } + ], + "expected": { + "sorted_paths": ["SKILL.md", "assets/empty.txt"], + "file_sha256": { + "SKILL.md": "7cfd32d4977363dda1f02aee644675610f1e928f3ab60647784f2f4c3fdc5ac8", + "assets/empty.txt": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "tree_hash": "46f64433798ebd6aac5a0330d73d22b24171dc1a4b8638251573df230578574a" + } + }, + { + "id": "binary-file", + "description": "Bytes that are not valid UTF-8, given as base64. Hash the raw bytes, never decoded text.", + "files": [ + { + "path": "SKILL.md", + "content_utf8": "---\nname: hello-world\ndescription: Say hello. Use when the user asks for a greeting.\n---\n\n# Hello World\n\nReply with one short greeting.\n" + }, + { + "path": "assets/blob.bin", + "content_base64": "AAF/gP8KDQA=" + } + ], + "expected": { + "sorted_paths": ["SKILL.md", "assets/blob.bin"], + "file_sha256": { + "SKILL.md": "7cfd32d4977363dda1f02aee644675610f1e928f3ab60647784f2f4c3fdc5ac8", + "assets/blob.bin": "1dc70ef8cde6b98f0b511be620f0e95df1385dc17f6078c9d65f15f0791933d7" + }, + "tree_hash": "a2e9d51d29f0db303346e98398bcf488aaf06707483e15f336338bc9ce53a6f6" + } + }, + { + "id": "ascii-case-order", + "description": "Uppercase sorts before lowercase in code unit order. Locale-aware compare (localeCompare, ICU collation) puts references/ and scripts/ first and gives a wrong hash. The shebang script also shows that the executable flag is not part of the hash.", + "files": [ + { + "path": "scripts/run.sh", + "content_utf8": "#!/bin/sh\necho hello\n" + }, + { + "path": "references/api.md", + "content_utf8": "# API\n\nGET /hello\n" + }, + { + "path": "Zebra.md", + "content_utf8": "Zebra notes.\n" + }, + { + "path": "SKILL.md", + "content_utf8": "---\nname: hello-world\ndescription: Say hello. Use when the user asks for a greeting.\n---\n\n# Hello World\n\nReply with one short greeting.\n" + } + ], + "expected": { + "sorted_paths": [ + "SKILL.md", + "Zebra.md", + "references/api.md", + "scripts/run.sh" + ], + "file_sha256": { + "SKILL.md": "7cfd32d4977363dda1f02aee644675610f1e928f3ab60647784f2f4c3fdc5ac8", + "Zebra.md": "24157d1dd113d657151458ac4e4ed849340d1636e90edf586134e54211c8f710", + "references/api.md": "1a7233bbabbbb08319b1d09499077bdaa2073786f5d0af249ae17e54af580a91", + "scripts/run.sh": "bfdeaeb08cffb6a36438bcd12dda25417e3cdd36f1e7e482a2849d539225288b" + }, + "tree_hash": "78edc54824986565c6a7dae6ce39880a265ab8e4d9a7a20618133498377af8b3" + } + }, + { + "id": "punctuation-and-space-order", + "description": "Code unit order of space, '-', '.', '/', and '_' is ' ' < '-' < '.' < '/' < '_'. The file docs.md sorts before the folder docs/.", + "files": [ + { + "path": "docs/guide.md", + "content_utf8": "Guide.\n" + }, + { + "path": "docs/API Guide.md", + "content_utf8": "API guide.\n" + }, + { + "path": "docs.md", + "content_utf8": "Docs.\n" + }, + { + "path": "a_b.md", + "content_utf8": "underscore\n" + }, + { + "path": "a.b.md", + "content_utf8": "dot\n" + }, + { + "path": "a-b.md", + "content_utf8": "dash\n" + }, + { + "path": "SKILL.md", + "content_utf8": "---\nname: hello-world\ndescription: Say hello. Use when the user asks for a greeting.\n---\n\n# Hello World\n\nReply with one short greeting.\n" + } + ], + "expected": { + "sorted_paths": [ + "SKILL.md", + "a-b.md", + "a.b.md", + "a_b.md", + "docs.md", + "docs/API Guide.md", + "docs/guide.md" + ], + "file_sha256": { + "SKILL.md": "7cfd32d4977363dda1f02aee644675610f1e928f3ab60647784f2f4c3fdc5ac8", + "a-b.md": "f8359416cedbf4b44bd1cab71b791b4121e3b33748187c530e70207af87c3f39", + "a.b.md": "5ddbce254c08372e429a250112c6f4593868687ab01e9a126193e5a83560362b", + "a_b.md": "10af960b268d98ed1e95acc1c1d7e9a655b0967acca44fdc4e4af2559ae05ce3", + "docs.md": "1fec416078e4d341cce91202bb2cf0416f4a3b84c211838086ef788c8154efd4", + "docs/API Guide.md": "bfb61b3d486e71d1113277bf7b856405d02c45f0950cac2bc7f93e87a853e67c", + "docs/guide.md": "2e8faaa5c331ce0718e04ada7cce3feaa5b675d2f44bbaf0a56b809a136f7563" + }, + "tree_hash": "5cf2c9d3d70943cce11bb26fe8854589bfdbc7e2a89c2d599ee6d98c9a39616f" + } + }, + { + "id": "unicode-nfc-path", + "description": "Path with \u00e9 as one code point (NFC, U+00E9).", + "files": [ + { + "path": "SKILL.md", + "content_utf8": "---\nname: hello-world\ndescription: Say hello. Use when the user asks for a greeting.\n---\n\n# Hello World\n\nReply with one short greeting.\n" + }, + { + "path": "notes/caf\u00e9.md", + "content_utf8": "Caf\u00e9 notes.\n" + } + ], + "expected": { + "sorted_paths": ["SKILL.md", "notes/caf\u00e9.md"], + "file_sha256": { + "SKILL.md": "7cfd32d4977363dda1f02aee644675610f1e928f3ab60647784f2f4c3fdc5ac8", + "notes/caf\u00e9.md": "1cb49a2aac2e4cafa0aa32fcafe98194ce3d1f98b7e5dccf9788baccf0b7b8dd" + }, + "tree_hash": "f94ebcd4a905687656d967e833da93f63ba1185c332d95d08dae6f9001b02428" + } + }, + { + "id": "unicode-nfd-path", + "description": "The same visible path as unicode-nfc-path, written as e plus a combining acute accent (NFD, U+0065 U+0301). No Unicode normalization is applied, so the tree hash differs from unicode-nfc-path.", + "files": [ + { + "path": "SKILL.md", + "content_utf8": "---\nname: hello-world\ndescription: Say hello. Use when the user asks for a greeting.\n---\n\n# Hello World\n\nReply with one short greeting.\n" + }, + { + "path": "notes/cafe\u0301.md", + "content_utf8": "Caf\u00e9 notes.\n" + } + ], + "expected": { + "sorted_paths": ["SKILL.md", "notes/cafe\u0301.md"], + "file_sha256": { + "SKILL.md": "7cfd32d4977363dda1f02aee644675610f1e928f3ab60647784f2f4c3fdc5ac8", + "notes/cafe\u0301.md": "1cb49a2aac2e4cafa0aa32fcafe98194ce3d1f98b7e5dccf9788baccf0b7b8dd" + }, + "tree_hash": "a568adafd0e9917d562a616c4348bf7ce479b3ab2c5f986c6237abd076d7bb9a" + } + }, + { + "id": "astral-vs-bmp-order", + "description": "U+1F600 (code units D83D DE00) sorts before U+FF5E in UTF-16 code unit order, but after it in code point or UTF-8 byte order. Use code unit order.", + "files": [ + { + "path": "SKILL.md", + "content_utf8": "---\nname: hello-world\ndescription: Say hello. Use when the user asks for a greeting.\n---\n\n# Hello World\n\nReply with one short greeting.\n" + }, + { + "path": "notes/\uff5e.md", + "content_utf8": "Fullwidth tilde.\n" + }, + { + "path": "notes/\ud83d\ude00.md", + "content_utf8": "Smile.\n" + } + ], + "expected": { + "sorted_paths": [ + "SKILL.md", + "notes/\ud83d\ude00.md", + "notes/\uff5e.md" + ], + "file_sha256": { + "SKILL.md": "7cfd32d4977363dda1f02aee644675610f1e928f3ab60647784f2f4c3fdc5ac8", + "notes/\ud83d\ude00.md": "80b766b7ddc31cbe62fbd7b38d4cde0b0dd5d9f9cfee610249aa2088bf401c88", + "notes/\uff5e.md": "bdbd7ad0693edfd96ba2202d25d05ba56aaa5b6ff30c0ab079be01ecc0b54025" + }, + "tree_hash": "6072afea213178dc29888c4274499d06c9ca95673258f343f79928a52d946c5d" + } + }, + { + "id": "nested-tree", + "description": "Several nested folders with text, code, and JSON files.", + "files": [ + { + "path": "SKILL.md", + "content_utf8": "---\nname: hello-world\ndescription: Say hello. Use when the user asks for a greeting.\n---\n\n# Hello World\n\nReply with one short greeting.\n" + }, + { + "path": "references/guides/setup/linux.md", + "content_utf8": "# Linux setup\n\nInstall the tool.\n" + }, + { + "path": "scripts/lib/util.py", + "content_utf8": "def greet(name):\n return f\"Hello, {name}!\"\n" + }, + { + "path": "assets/templates/report.json", + "content_utf8": "{\n \"title\": \"Report\"\n}\n" + } + ], + "expected": { + "sorted_paths": [ + "SKILL.md", + "assets/templates/report.json", + "references/guides/setup/linux.md", + "scripts/lib/util.py" + ], + "file_sha256": { + "SKILL.md": "7cfd32d4977363dda1f02aee644675610f1e928f3ab60647784f2f4c3fdc5ac8", + "assets/templates/report.json": "212b800b433da09a20af7ee21993b5fbed35c20c5a740cefd2a359a10c207f57", + "references/guides/setup/linux.md": "4b7b65327eeffd38cc735dc0fd07a1b1f27069e47532825c5314297c514f7a3d", + "scripts/lib/util.py": "a55c14346b5c0e11da873793db175efd28d4840a330a5edba18d03d7d7310626" + }, + "tree_hash": "27dd61f18b581e9c6be7cfe78dd7ef2ce7dbb811d55bd7ddac54725abe9bb827" + } + } + ], + "invalid_paths": [ + { + "path": "", + "reason": "empty path" + }, + { + "path": "../escape.md", + "reason": "dot segment" + }, + { + "path": "a/../b.md", + "reason": "dot segment" + }, + { + "path": "./a.md", + "reason": "dot segment" + }, + { + "path": "a/./b.md", + "reason": "dot segment" + }, + { + "path": "/etc/passwd", + "reason": "absolute path" + }, + { + "path": "C:/Windows/x.md", + "reason": "absolute path with a drive letter" + }, + { + "path": "a//b.md", + "reason": "empty segment" + }, + { + "path": "dir/", + "reason": "empty segment (trailing slash)" + } + ] +} diff --git a/packages/agentvouch-protocol/test-vectors/tree_hash_reference.py b/packages/agentvouch-protocol/test-vectors/tree_hash_reference.py new file mode 100644 index 00000000..395ea3fd --- /dev/null +++ b/packages/agentvouch-protocol/test-vectors/tree_hash_reference.py @@ -0,0 +1,231 @@ +#!/usr/bin/env python3 +"""Reference implementation of the AgentVouch skill tree hash, version 1. + +Algorithm id: agentvouch-tree-v1. Spec: docs/SKILL_TREE_HASH.md. + +This file is independent of web/lib/skillStorage.ts on purpose: two +implementations in two languages check the same vectors. Standard library only. + +Usage (paths relative to the repository root): + + python3 packages/agentvouch-protocol/test-vectors/tree_hash_reference.py + Check every vector in tree-hash.json. Exit 1 on any mismatch. + + python3 packages/agentvouch-protocol/test-vectors/tree_hash_reference.py --write + Recompute every "expected" block from the inputs. Review the diff, then + run prettier on tree-hash.json before you commit it. + + python3 packages/agentvouch-protocol/test-vectors/tree_hash_reference.py --dir PATH + Print the tree hash of a local skill folder. Uses the CLI publish skip + rules: .git and node_modules folders, .DS_Store files, *.agentvouch.json + files, and symbolic links are left out. +""" + +import argparse +import base64 +import hashlib +import json +import os +import re +import sys + +ALGORITHM = "agentvouch-tree-v1" +VECTORS_SCHEMA = "agentvouch.tree-hash-vectors/v1" +VECTORS_PATH = os.path.join( + os.path.dirname(os.path.abspath(__file__)), "tree-hash.json" +) + +_DRIVE_LETTER = re.compile(r"^[A-Za-z]:/") +_SKIP_DIRECTORY_NAMES = {".git", "node_modules"} +_SKIP_FILE_NAMES = {".DS_Store"} + + +def file_digest(data): + """Lowercase hex SHA-256 of the exact file bytes.""" + return hashlib.sha256(data).hexdigest() + + +def _utf16_key(text): + # Big-endian UTF-16 bytes compare in the same order as UTF-16 code units. + # That is the JavaScript default string sort used by computeTreeHash. + return text.encode("utf-16-be", "surrogatepass") + + +def sort_paths(paths): + """Sort paths in UTF-16 code unit order (JavaScript default sort).""" + return sorted(paths, key=_utf16_key) + + +def tree_hash_from_digests(digests): + """digests: iterable of (path, lowercase hex sha256). Returns the tree hash.""" + entries = ["%s\0%s" % (path, digest) for path, digest in digests] + entries.sort(key=_utf16_key) + return hashlib.sha256("\n".join(entries).encode("utf-8")).hexdigest() + + +def tree_hash(files): + """files: iterable of (path, bytes). Returns the tree hash.""" + return tree_hash_from_digests( + (path, file_digest(data)) for path, data in files + ) + + +def path_error(path): + """Return why a stored path breaks the path rules, or None if it is valid.""" + if not path: + return "empty path" + if path != path.strip() or path.startswith("") or path.endswith(""): + return "leading or trailing white space" + if "\\" in path: + return "backslash (stored paths use / only)" + if path.startswith("/") or _DRIVE_LETTER.match(path): + return "absolute path" + segments = path.split("/") + if any(segment == "" for segment in segments): + return "empty segment" + if any(segment in (".", "..") for segment in segments): + return "dot segment" + return None + + +def tree_errors(paths): + """Return tree-level errors: duplicates, SKILL.md, and file/folder conflicts.""" + errors = set() + seen = set() + for path in paths: + if path in seen: + errors.add("duplicate path: %s" % path) + seen.add(path) + if "SKILL.md" not in seen: + errors.add("missing top-level SKILL.md") + for path in seen: + parts = path.split("/") + for index in range(1, len(parts)): + prefix = "/".join(parts[:index]) + if prefix in seen: + errors.add("path conflict: %s is a file and a folder" % prefix) + return sorted(errors) + + +def _vector_bytes(file_entry): + if "content_base64" in file_entry: + return base64.b64decode(file_entry["content_base64"], validate=True) + return file_entry["content_utf8"].encode("utf-8") + + +def compute_expected(vector): + files = [(entry["path"], _vector_bytes(entry)) for entry in vector["files"]] + problems = [ + "%s: %s" % (path, path_error(path)) for path, _ in files if path_error(path) + ] + problems.extend(tree_errors([path for path, _ in files])) + if problems: + raise ValueError( + "vector %s is not a valid tree: %s" % (vector["id"], "; ".join(problems)) + ) + ordered = sort_paths([path for path, _ in files]) + digests = {path: file_digest(data) for path, data in files} + return { + "sorted_paths": ordered, + "file_sha256": {path: digests[path] for path in ordered}, + "tree_hash": tree_hash(files), + } + + +def load_vectors(path=VECTORS_PATH): + with open(path, encoding="utf-8") as handle: + document = json.load(handle) + if document.get("schema") != VECTORS_SCHEMA: + raise ValueError("unexpected vectors schema: %r" % document.get("schema")) + if document.get("algorithm") != ALGORITHM: + raise ValueError("unexpected algorithm: %r" % document.get("algorithm")) + return document + + +def check_vectors(document): + failures = [] + ids = [vector["id"] for vector in document["vectors"]] + if len(ids) != len(set(ids)): + failures.append("duplicate vector ids") + for vector in document["vectors"]: + try: + actual = compute_expected(vector) + except ValueError as error: + failures.append(str(error)) + continue + if actual != vector.get("expected"): + failures.append("vector %s: expected block does not match" % vector["id"]) + for case in document.get("invalid_paths", []): + if path_error(case["path"]) is None: + failures.append("invalid path %r was accepted" % case["path"]) + return failures + + +def write_vectors(document, path=VECTORS_PATH): + for vector in document["vectors"]: + vector["expected"] = compute_expected(vector) + with open(path, "w", encoding="utf-8") as handle: + # ensure_ascii keeps non-ASCII paths as \u escapes, so editors and tools + # that normalize Unicode cannot silently change the NFC/NFD vectors. + handle.write(json.dumps(document, indent=2, ensure_ascii=True) + "\n") + + +def collect_directory(root): + files = [] + for current, folders, names in os.walk(root, followlinks=False): + folders[:] = sorted( + name + for name in folders + if name not in _SKIP_DIRECTORY_NAMES + and not os.path.islink(os.path.join(current, name)) + ) + for name in names: + absolute = os.path.join(current, name) + if os.path.islink(absolute): + continue + if name in _SKIP_FILE_NAMES or name.endswith(".agentvouch.json"): + continue + relative = os.path.relpath(absolute, root).replace(os.sep, "/") + with open(absolute, "rb") as handle: + files.append((relative, handle.read())) + return files + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + group = parser.add_mutually_exclusive_group() + group.add_argument("--write", action="store_true", help="recompute expected blocks") + group.add_argument("--dir", help="print the tree hash of a local skill folder") + args = parser.parse_args(argv) + + if args.dir: + files = collect_directory(args.dir) + problems = [ + "%s: %s" % (path, path_error(path)) for path, _ in files if path_error(path) + ] + problems.extend(tree_errors([path for path, _ in files])) + for problem in problems: + print("warning: %s" % problem, file=sys.stderr) + print(tree_hash(files)) + return 1 if problems else 0 + + document = load_vectors() + if args.write: + write_vectors(document) + print("wrote %d vectors to %s" % (len(document["vectors"]), VECTORS_PATH)) + return 0 + + failures = check_vectors(document) + for failure in failures: + print("FAIL %s" % failure) + if failures: + return 1 + print( + "ok: %d vectors and %d invalid paths match %s" + % (len(document["vectors"]), len(document.get("invalid_paths", [])), ALGORITHM) + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/web/__tests__/lib/treeHashAudit.test.ts b/web/__tests__/lib/treeHashAudit.test.ts new file mode 100644 index 00000000..346005a9 --- /dev/null +++ b/web/__tests__/lib/treeHashAudit.test.ts @@ -0,0 +1,346 @@ +import { createHash } from "crypto"; +import { neon } from "@neondatabase/serverless"; +import { describe, expect, it } from "vitest"; +import { buildSkillManifest, computeTreeHash } from "@/lib/skillStorage"; +import { + auditSkillVersionTree, + buildSweepVersionQuery, + findTreePathProblems, + parseTreeManifest, + sweepRowFromVersionRecord, + sweepTreeHashes, + type TreeFile, + type TreeHashAuditInput, + type TreeHashSweepRow, +} from "@/lib/treeHashAudit"; + +const SKILL_MD = "---\nname: demo\ndescription: Demo skill.\n---\n\n# Demo\n"; + +function sha256(bytes: Buffer): string { + return createHash("sha256").update(bytes).digest("hex"); +} + +function tree(files: Record): TreeFile[] { + return Object.entries(files).map(([path, text]) => ({ + path, + bytes: Buffer.from(text, "utf8"), + })); +} + +/** A stored version whose manifest, tree_hash, and content all agree. */ +function cleanVersion(files: TreeFile[]): TreeHashAuditInput { + const manifest = buildSkillManifest(files); + const skillMd = files.find((file) => file.path === "SKILL.md"); + return { + storedTreeHash: computeTreeHash(manifest), + manifest, + content: skillMd ? skillMd.bytes.toString("utf8") : null, + files, + }; +} + +function codes(input: TreeHashAuditInput): string[] { + return auditSkillVersionTree(input) + .findings.map((finding) => finding.code) + .sort(); +} + +const MULTI_FILE = tree({ + "SKILL.md": SKILL_MD, + "references/api.md": "# API\n", + "scripts/run.sh": "#!/bin/sh\necho hi\n", +}); + +describe("auditSkillVersionTree", () => { + it("passes a clean single-file (inline) version", () => { + const result = auditSkillVersionTree( + cleanVersion(tree({ "SKILL.md": SKILL_MD })) + ); + expect(result.ok).toBe(true); + expect(result.findings).toEqual([]); + expect(result.treeHashFromBytes).toBe(result.treeHashFromManifest); + }); + + it("passes a clean multi-file version and accepts the manifest as JSON text", () => { + const input = cleanVersion(MULTI_FILE); + expect(auditSkillVersionTree(input).ok).toBe(true); + expect( + auditSkillVersionTree({ + ...input, + manifest: JSON.stringify(input.manifest), + }).ok + ).toBe(true); + }); + + it("reports a stored tree_hash that matches neither the manifest nor the bytes", () => { + const input = cleanVersion(MULTI_FILE); + expect(codes({ ...input, storedTreeHash: "0".repeat(64) })).toEqual([ + "bytes_tree_hash_mismatch", + "manifest_tree_hash_mismatch", + ]); + }); + + it("reports missing and malformed tree hashes without other noise", () => { + const input = cleanVersion(MULTI_FILE); + expect(codes({ ...input, storedTreeHash: null })).toEqual([ + "missing_tree_hash", + ]); + expect(codes({ ...input, storedTreeHash: "ABC" })).toEqual([ + "malformed_tree_hash", + ]); + }); + + it("reports stored bytes that changed after the manifest was written", () => { + const input = cleanVersion(MULTI_FILE); + const changed = MULTI_FILE.map((file) => + file.path === "references/api.md" + ? { ...file, bytes: Buffer.from("# API v2\n", "utf8") } + : file + ); + const result = auditSkillVersionTree({ ...input, files: changed }); + expect(result.findings.map((finding) => finding.code).sort()).toEqual([ + "bytes_tree_hash_mismatch", + "file_digest_mismatch", + "file_size_mismatch", + ]); + expect( + result.findings.find((finding) => finding.code === "file_digest_mismatch") + ?.path + ).toBe("references/api.md"); + }); + + it("reports a manifest digest that does not match the bytes", () => { + const input = cleanVersion(MULTI_FILE); + const manifest = buildSkillManifest(MULTI_FILE).map((entry) => + entry.path === "scripts/run.sh" + ? { ...entry, sha256: sha256(Buffer.from("other")) } + : entry + ); + expect(codes({ ...input, manifest })).toEqual([ + "file_digest_mismatch", + "manifest_tree_hash_mismatch", + ]); + }); + + it("reports a manifest size that does not match the bytes", () => { + const input = cleanVersion(MULTI_FILE); + const manifest = buildSkillManifest(MULTI_FILE).map((entry) => + entry.path === "SKILL.md" ? { ...entry, size: entry.size + 1 } : entry + ); + expect(codes({ ...input, manifest })).toEqual(["file_size_mismatch"]); + }); + + it("reports files that are only in storage or only in the manifest", () => { + const input = cleanVersion(MULTI_FILE); + const extra = [ + ...MULTI_FILE, + { path: "assets/extra.txt", bytes: Buffer.from("x") }, + ]; + expect(codes({ ...input, files: extra })).toEqual([ + "bytes_tree_hash_mismatch", + "file_missing_from_manifest", + ]); + + const missing = MULTI_FILE.filter((file) => file.path !== "scripts/run.sh"); + expect(codes({ ...input, files: missing })).toEqual([ + "bytes_tree_hash_mismatch", + "file_missing_from_storage", + ]); + }); + + it("reports when /raw content differs from SKILL.md in the stored tree", () => { + const input = cleanVersion(MULTI_FILE); + expect(codes({ ...input, content: `${SKILL_MD}\nextra\n` })).toEqual([ + "raw_skill_md_mismatch", + ]); + }); + + it("still checks the manifest when stored files cannot be loaded", () => { + const input = cleanVersion(MULTI_FILE); + const result = auditSkillVersionTree({ + ...input, + files: null, + loadError: "Stored skill archive was not found", + }); + expect(result.findings.map((finding) => finding.code)).toEqual([ + "files_unavailable", + ]); + expect(result.findings[0].detail).toContain( + "Stored skill archive was not found" + ); + expect(result.treeHashFromManifest).toBe(input.storedTreeHash); + expect(result.treeHashFromBytes).toBeNull(); + }); + + it("reports missing and invalid manifests", () => { + const input = cleanVersion(MULTI_FILE); + expect(codes({ ...input, manifest: null })).toEqual(["missing_manifest"]); + expect(codes({ ...input, manifest: "{not json" })).toEqual([ + "invalid_manifest", + ]); + expect(codes({ ...input, manifest: { path: "SKILL.md" } })).toEqual([ + "invalid_manifest", + ]); + }); +}); + +describe("parseTreeManifest", () => { + it("rejects entries without a lowercase hex sha256 or a valid size", () => { + const good = { path: "SKILL.md", size: 1, sha256: "a".repeat(64) }; + expect(parseTreeManifest([good]).entries).toEqual([good]); + expect( + parseTreeManifest([{ ...good, sha256: "A".repeat(64) }]).error + ).toBeDefined(); + expect(parseTreeManifest([{ ...good, size: -1 }]).error).toBeDefined(); + expect(parseTreeManifest([{ ...good, size: 1.5 }]).error).toBeDefined(); + expect(parseTreeManifest(undefined)).toEqual({ entries: null }); + }); +}); + +describe("findTreePathProblems", () => { + it("accepts a normal tree", () => { + expect( + findTreePathProblems(["SKILL.md", "references/a.md", "scripts/run.sh"]) + ).toEqual([]); + }); + + it("reports a path that is both a file and a folder", () => { + const findings = findTreePathProblems(["SKILL.md", "docs", "docs/a.md"]); + expect(findings.map((finding) => finding.code)).toEqual(["path_conflict"]); + expect(findings[0].path).toBe("docs"); + }); + + it("reports duplicates, a missing SKILL.md, and paths not in stored form", () => { + expect( + findTreePathProblems(["README.md", "README.md"]).map( + (finding) => finding.code + ) + ).toEqual(["duplicate_path", "missing_skill_md"]); + expect( + findTreePathProblems(["SKILL.md", "a\\b.md", " c.md", "../d.md"]).map( + (finding) => finding.code + ) + ).toEqual(["invalid_path", "invalid_path", "invalid_path"]); + }); +}); + +describe("sweepTreeHashes", () => { + function row(id: string, input: TreeHashAuditInput): TreeHashSweepRow { + return { + versionId: `${id}-version`, + skillId: id, + version: 1, + label: `author/${id} v1`, + storedTreeHash: input.storedTreeHash, + manifest: input.manifest, + storageBackend: "blob", + content: input.content, + }; + } + + it("audits every row, counts findings by code, and keeps going after load errors", async () => { + const clean = cleanVersion(MULTI_FILE); + const rows = [ + row("clean", clean), + row("tampered", { ...clean, storedTreeHash: "f".repeat(64) }), + row("unloadable", clean), + ]; + const seen: string[] = []; + const summary = await sweepTreeHashes( + rows, + async (current) => { + if (current.skillId === "unloadable") throw new Error("blob missing"); + return MULTI_FILE; + }, + (entry) => seen.push(`${entry.row.skillId}:${entry.result.ok}`) + ); + + expect(seen).toEqual(["clean:true", "tampered:false", "unloadable:false"]); + expect(summary.checked).toBe(3); + expect(summary.ok).toBe(1); + expect(summary.withFindings).toBe(2); + expect(summary.findingsByCode).toEqual({ + bytes_tree_hash_mismatch: 1, + manifest_tree_hash_mismatch: 1, + files_unavailable: 1, + }); + expect(summary.failures.map((entry) => entry.row.skillId)).toEqual([ + "tampered", + "unloadable", + ]); + }); +}); + +describe("buildSweepVersionQuery", () => { + // neon() builds queries lazily: nothing is sent until a query is awaited, so + // these tests compile the exact SQL offline. + const db = neon( + "postgresql://user:password@ep-example.us-east-2.aws.neon.tech/db" + ); + + function compile(options: Parameters[1]) { + const query = buildSweepVersionQuery(db, options) as unknown as { + queryData: { + toParameterizedQuery(): { query: string; params: unknown[] }; + }; + }; + return query.queryData.toParameterizedQuery(); + } + + it.each([ + { latestOnly: false, skillId: null, limit: null }, + { latestOnly: true, skillId: null, limit: null }, + { + latestOnly: true, + skillId: "11111111-2222-4333-8444-555555555555", + limit: 25, + }, + ])("is a single SELECT with no write statements (%o)", (options) => { + const { query } = compile(options); + const normalized = query.replace(/\s+/g, " ").trim(); + expect(normalized.startsWith("SELECT")).toBe(true); + expect(normalized).not.toMatch( + /\b(INSERT|UPDATE|DELETE|ALTER|CREATE|DROP|TRUNCATE|GRANT|COPY|MERGE)\b/i + ); + expect(normalized).not.toContain(";"); + }); + + it("binds the skill filter and the limit as parameters", () => { + const skillId = "11111111-2222-4333-8444-555555555555"; + const { query, params } = compile({ + latestOnly: false, + skillId, + limit: 25, + }); + expect(params).toEqual([skillId, 25]); + expect(query).toContain("AND v.skill_id = $1::uuid"); + expect(query).toContain("LIMIT $2"); + expect(query).not.toContain(skillId); + }); + + it("selects the newest version per skill in latest-only mode", () => { + const { query } = compile({ latestOnly: true, skillId: null, limit: null }); + expect(query).toContain("DISTINCT ON (v.skill_id)"); + expect(query).toContain("ORDER BY v.skill_id, v.version DESC"); + }); +}); + +describe("sweepRowFromVersionRecord", () => { + it("labels rows with the public route when it exists", () => { + const base = { + version_id: "v-1", + skill_id: "s-1", + version: 3, + content: SKILL_MD, + files: [], + tree_hash: "a".repeat(64), + storage_backend: "inline", + public_author_slug: "anthropics", + public_slug: "pdf", + }; + expect(sweepRowFromVersionRecord(base).label).toBe("anthropics/pdf v3"); + expect( + sweepRowFromVersionRecord({ ...base, public_slug: null }).label + ).toBe("s-1 v3"); + }); +}); diff --git a/web/__tests__/lib/treeHashVectors.test.ts b/web/__tests__/lib/treeHashVectors.test.ts new file mode 100644 index 00000000..a9052618 --- /dev/null +++ b/web/__tests__/lib/treeHashVectors.test.ts @@ -0,0 +1,172 @@ +import { createHash } from "crypto"; +import { readFileSync } from "fs"; +import path from "path"; +import { describe, expect, it } from "vitest"; +import { + buildSkillManifest, + computeTreeHash, + normalizeSkillPath, + prepareSkillTree, +} from "@/lib/skillStorage"; + +// Shared vectors for agentvouch-tree-v1 (docs/SKILL_TREE_HASH.md). The expected +// values come from an independent stdlib Python implementation +// (packages/agentvouch-protocol/test-vectors/tree_hash_reference.py), so this +// test checks the production code against a second implementation. + +type VectorFile = { + path: string; + content_utf8?: string; + content_base64?: string; +}; + +type Vector = { + id: string; + description: string; + files: VectorFile[]; + expected: { + sorted_paths: string[]; + file_sha256: Record; + tree_hash: string; + }; +}; + +type VectorDocument = { + schema: string; + algorithm: string; + vectors: Vector[]; + invalid_paths: Array<{ path: string; reason: string }>; +}; + +const VECTORS_PATH = path.resolve( + __dirname, + "../../../packages/agentvouch-protocol/test-vectors/tree-hash.json" +); +const document: VectorDocument = JSON.parse(readFileSync(VECTORS_PATH, "utf8")); + +function fileBytes(file: VectorFile): Buffer { + if (file.content_base64 !== undefined) { + return Buffer.from(file.content_base64, "base64"); + } + if (file.content_utf8 !== undefined) { + return Buffer.from(file.content_utf8, "utf8"); + } + throw new Error(`Vector file ${file.path} has no content`); +} + +function filesOf(vector: Vector) { + return vector.files.map((file) => ({ + path: file.path, + bytes: fileBytes(file), + })); +} + +function vectorById(id: string): Vector { + const vector = document.vectors.find((candidate) => candidate.id === id); + if (!vector) throw new Error(`Missing vector ${id}`); + return vector; +} + +describe("tree hash shared vectors (agentvouch-tree-v1)", () => { + it("uses the documented schema and algorithm id", () => { + expect(document.schema).toBe("agentvouch.tree-hash-vectors/v1"); + expect(document.algorithm).toBe("agentvouch-tree-v1"); + expect(document.vectors.length).toBeGreaterThanOrEqual(10); + expect(new Set(document.vectors.map((vector) => vector.id)).size).toBe( + document.vectors.length + ); + }); + + describe.each(document.vectors.map((vector) => [vector.id, vector] as const))( + "%s", + (_id, vector) => { + it("matches every file digest", () => { + const manifest = buildSkillManifest(filesOf(vector)); + expect( + Object.fromEntries( + manifest.map((entry) => [entry.path, entry.sha256]) + ) + ).toEqual(vector.expected.file_sha256); + }); + + it("matches the tree hash computed from file bytes", () => { + expect(computeTreeHash(filesOf(vector))).toBe( + vector.expected.tree_hash + ); + }); + + it("matches the tree hash computed from manifest digests alone", () => { + const digestsOnly = Object.entries(vector.expected.file_sha256).map( + ([filePath, sha256]) => ({ path: filePath, sha256 }) + ); + expect(computeTreeHash(digestsOnly)).toBe(vector.expected.tree_hash); + }); + + it("orders paths by UTF-16 code units (JavaScript default sort)", () => { + expect(vector.files.map((file) => file.path).sort()).toEqual( + vector.expected.sorted_paths + ); + }); + + it("keeps the paths and the tree hash through the publish path", () => { + const tree = prepareSkillTree( + filesOf(vector).map((file) => ({ + path: file.path, + content: file.bytes, + })) + ); + expect(tree.treeHash).toBe(vector.expected.tree_hash); + expect(tree.manifest.map((entry) => entry.path).sort()).toEqual( + vector.expected.sorted_paths + ); + }); + } + ); + + it("matches the single-file SQL backfill formula in web/lib/db.ts", () => { + const single = vectorById("single-skill-md"); + const digest = single.expected.file_sha256["SKILL.md"]; + // db.ts: digest('SKILL.md' || '\x00' || , 'sha256') + const sqlEquivalent = createHash("sha256") + .update( + Buffer.concat([ + Buffer.from("SKILL.md", "utf8"), + Buffer.from([0]), + Buffer.from(digest, "utf8"), + ]) + ) + .digest("hex"); + expect(sqlEquivalent).toBe(single.expected.tree_hash); + }); + + it("hashes bytes exactly: CRLF and LF versions differ", () => { + expect(vectorById("crlf-line-endings").expected.tree_hash).not.toBe( + vectorById("single-skill-md").expected.tree_hash + ); + }); + + it("applies no Unicode normalization to paths", () => { + const nfc = vectorById("unicode-nfc-path"); + const nfd = vectorById("unicode-nfd-path"); + expect(nfc.files[1].path.normalize("NFC")).toBe( + nfd.files[1].path.normalize("NFC") + ); + expect(nfc.files[1].path).not.toBe(nfd.files[1].path); + expect(nfc.expected.tree_hash).not.toBe(nfd.expected.tree_hash); + }); + + it("does not use locale-aware ordering", () => { + const vector = vectorById("ascii-case-order"); + const localeOrder = vector.files + .map((file) => file.path) + .sort((a, b) => a.localeCompare(b)); + expect(localeOrder).not.toEqual(vector.expected.sorted_paths); + }); + + it.each(document.invalid_paths.map((entry) => [entry.path, entry.reason]))( + "rejects invalid path %j (%s)", + (invalidPath) => { + expect(() => normalizeSkillPath(invalidPath)).toThrow(); + } + ); +}); diff --git a/web/__tests__/scripts/tree-hash-sweep-source.test.ts b/web/__tests__/scripts/tree-hash-sweep-source.test.ts new file mode 100644 index 00000000..9b1684f7 --- /dev/null +++ b/web/__tests__/scripts/tree-hash-sweep-source.test.ts @@ -0,0 +1,42 @@ +import { readFileSync } from "fs"; +import { join } from "path"; +import { describe, expect, it } from "vitest"; + +// The sweep must stay read-only. Its one query is built (and tested) in +// lib/treeHashAudit.ts; these checks guard the I/O wiring around it. +describe("tree-hash sweep script source", () => { + const source = readFileSync( + join(process.cwd(), "scripts", "tree-hash-sweep.ts"), + "utf8" + ); + const code = source + .replace(/\/\*[\s\S]*?\*\//g, "") + .replace(/^\s*\/\/.*$/gm, ""); + + it("reads rows only through the shared SELECT-only query builder", () => { + expect(code).toContain("buildSweepVersionQuery("); + expect(code).not.toMatch( + /\b(INSERT|UPDATE|DELETE|ALTER|CREATE|DROP|TRUNCATE)\b/ + ); + }); + + it("never runs schema setup or imports the app database module", () => { + expect(code).not.toContain("initializeDatabase"); + expect(code).not.toContain('"@/lib/db"'); + }); + + it("never writes to Blob storage", () => { + expect(code).not.toContain('"@vercel/blob"'); + expect(code).not.toMatch(/\b(put|del|copy)\s*\(/); + }); + + it("does not call the public download routes, which record download events", () => { + expect(code).not.toMatch(/\bfetch\s*\(/); + expect(code).not.toContain("/api/skills/"); + }); + + it("prints the target host and honors EXPECTED_DATABASE_HOST", () => { + expect(code).toContain("Database host:"); + expect(code).toContain("EXPECTED_DATABASE_HOST"); + }); +}); diff --git a/web/lib/treeHashAudit.ts b/web/lib/treeHashAudit.ts new file mode 100644 index 00000000..d8996ffd --- /dev/null +++ b/web/lib/treeHashAudit.ts @@ -0,0 +1,476 @@ +// Tree-hash audit for stored skill versions (algorithm agentvouch-tree-v1). +// +// Pure functions: callers load the stored row and the file bytes, this module +// compares them. The read-only sweep (scripts/tree-hash-sweep.ts) uses it to +// check that every stored version's bytes, manifest, and tree_hash agree, and +// that /api/skills/{id}/raw (which serves skill_versions.content for SKILL.md) +// returns the same SKILL.md bytes as the tree. Spec: docs/SKILL_TREE_HASH.md. + +import { createHash } from "crypto"; +import { computeTreeHash, normalizeSkillPath } from "@/lib/skillStorage"; + +export const TREE_HASH_ALGORITHM = "agentvouch-tree-v1"; + +export type TreeHashFindingCode = + | "missing_tree_hash" + | "malformed_tree_hash" + | "missing_manifest" + | "invalid_manifest" + | "manifest_tree_hash_mismatch" + | "files_unavailable" + | "bytes_tree_hash_mismatch" + | "file_digest_mismatch" + | "file_size_mismatch" + | "file_missing_from_manifest" + | "file_missing_from_storage" + | "raw_skill_md_mismatch" + | "invalid_path" + | "duplicate_path" + | "missing_skill_md" + | "path_conflict"; + +export interface TreeHashFinding { + code: TreeHashFindingCode; + detail: string; + path?: string; +} + +export interface TreeManifestEntry { + path: string; + size: number; + sha256: string; +} + +export interface TreeFile { + path: string; + bytes: Buffer; +} + +export interface TreeHashAuditInput { + /** skill_versions.tree_hash */ + storedTreeHash: string | null; + /** skill_versions.files, as read from the database (JSONB array or JSON text). */ + manifest: unknown; + /** skill_versions.content. /api/skills/{id}/raw serves it for SKILL.md. */ + content: string | null; + /** Files loaded from storage, or null when they could not be loaded. */ + files: ReadonlyArray | null; + /** Why the files could not be loaded, when files is null. */ + loadError?: string | null; +} + +export interface TreeHashAuditResult { + ok: boolean; + findings: TreeHashFinding[]; + treeHashFromManifest: string | null; + treeHashFromBytes: string | null; +} + +const HEX64 = /^[0-9a-f]{64}$/; + +function sha256Hex(bytes: Buffer): string { + return createHash("sha256").update(bytes).digest("hex"); +} + +export function parseTreeManifest(value: unknown): { + entries: TreeManifestEntry[] | null; + error?: string; +} { + let parsed: unknown = value; + if (typeof value === "string") { + try { + parsed = JSON.parse(value); + } catch { + return { entries: null, error: "manifest is not valid JSON" }; + } + } + if (parsed === null || parsed === undefined) return { entries: null }; + if (!Array.isArray(parsed)) { + return { entries: null, error: "manifest is not an array" }; + } + + const entries: TreeManifestEntry[] = []; + for (const [index, item] of parsed.entries()) { + const record = + item && typeof item === "object" + ? (item as Record) + : null; + const path = record?.path; + const size = record?.size; + const sha256 = record?.sha256; + if ( + typeof path !== "string" || + typeof sha256 !== "string" || + !HEX64.test(sha256) || + typeof size !== "number" || + !Number.isInteger(size) || + size < 0 + ) { + return { + entries: null, + error: `manifest entry ${index} needs a path, a non-negative integer size, and a lowercase hex sha256`, + }; + } + entries.push({ path, size, sha256 }); + } + return { entries }; +} + +/** + * Tree-level path checks from docs/SKILL_TREE_HASH.md: stored paths are already + * in normalized form, unique, include one top-level SKILL.md, and no path is + * also a folder of another path (such a tree cannot be written to disk). + */ +export function findTreePathProblems( + paths: readonly string[] +): TreeHashFinding[] { + const findings: TreeHashFinding[] = []; + const seen = new Set(); + + for (const path of paths) { + if (seen.has(path)) { + findings.push({ + code: "duplicate_path", + path, + detail: `Path appears more than once: ${JSON.stringify(path)}`, + }); + continue; + } + seen.add(path); + + let normalized: string | null = null; + try { + normalized = normalizeSkillPath(path); + } catch (error) { + findings.push({ + code: "invalid_path", + path, + detail: (error as Error).message, + }); + continue; + } + if (normalized !== path) { + findings.push({ + code: "invalid_path", + path, + detail: `Stored path is not in normalized form (normalizes to ${JSON.stringify( + normalized + )})`, + }); + } + } + + if (!seen.has("SKILL.md")) { + findings.push({ + code: "missing_skill_md", + detail: "Tree has no top-level SKILL.md", + }); + } + + for (const path of seen) { + const parts = path.split("/"); + for (let index = 1; index < parts.length; index += 1) { + const prefix = parts.slice(0, index).join("/"); + if (seen.has(prefix)) { + findings.push({ + code: "path_conflict", + path: prefix, + detail: `${JSON.stringify( + prefix + )} is a file and also a folder of ${JSON.stringify(path)}`, + }); + } + } + } + + return findings; +} + +export function auditSkillVersionTree( + input: TreeHashAuditInput +): TreeHashAuditResult { + const findings: TreeHashFinding[] = []; + const stored = input.storedTreeHash; + + if (!stored) { + findings.push({ + code: "missing_tree_hash", + detail: "Version has no stored tree_hash", + }); + } else if (!HEX64.test(stored)) { + findings.push({ + code: "malformed_tree_hash", + detail: "Stored tree_hash is not 64 lowercase hex characters", + }); + } + const comparableStored = stored && HEX64.test(stored) ? stored : null; + + const manifest = parseTreeManifest(input.manifest); + let treeHashFromManifest: string | null = null; + if (manifest.error) { + findings.push({ code: "invalid_manifest", detail: manifest.error }); + } else if (!manifest.entries) { + findings.push({ + code: "missing_manifest", + detail: "Version has no stored file manifest", + }); + } else { + treeHashFromManifest = computeTreeHash(manifest.entries); + if (comparableStored && treeHashFromManifest !== comparableStored) { + findings.push({ + code: "manifest_tree_hash_mismatch", + detail: `Manifest digests give ${treeHashFromManifest}, stored tree_hash is ${comparableStored}`, + }); + } + } + + let treeHashFromBytes: string | null = null; + if (!input.files) { + findings.push({ + code: "files_unavailable", + detail: input.loadError + ? `Could not load stored files: ${input.loadError}` + : "Could not load stored files", + }); + if (manifest.entries) { + findings.push( + ...findTreePathProblems(manifest.entries.map((entry) => entry.path)) + ); + } + } else { + findings.push( + ...findTreePathProblems(input.files.map((file) => file.path)) + ); + + treeHashFromBytes = computeTreeHash([...input.files]); + if (comparableStored && treeHashFromBytes !== comparableStored) { + findings.push({ + code: "bytes_tree_hash_mismatch", + detail: `Stored bytes give ${treeHashFromBytes}, stored tree_hash is ${comparableStored}`, + }); + } + + if (manifest.entries) { + const manifestByPath = new Map( + manifest.entries.map((entry) => [entry.path, entry]) + ); + const storedPaths = new Set(); + for (const file of input.files) { + storedPaths.add(file.path); + const entry = manifestByPath.get(file.path); + if (!entry) { + findings.push({ + code: "file_missing_from_manifest", + path: file.path, + detail: "Stored file has no manifest entry", + }); + continue; + } + const digest = sha256Hex(file.bytes); + if (digest !== entry.sha256) { + findings.push({ + code: "file_digest_mismatch", + path: file.path, + detail: `Stored bytes hash to ${digest}, manifest says ${entry.sha256}`, + }); + } + if (file.bytes.byteLength !== entry.size) { + findings.push({ + code: "file_size_mismatch", + path: file.path, + detail: `Stored file is ${file.bytes.byteLength} bytes, manifest says ${entry.size}`, + }); + } + } + for (const entry of manifest.entries) { + if (!storedPaths.has(entry.path)) { + findings.push({ + code: "file_missing_from_storage", + path: entry.path, + detail: "Manifest lists a file that storage does not have", + }); + } + } + } + + if (input.content !== null) { + const treeSkillMd = input.files.find((file) => file.path === "SKILL.md"); + if ( + treeSkillMd && + !Buffer.from(input.content, "utf8").equals(treeSkillMd.bytes) + ) { + findings.push({ + code: "raw_skill_md_mismatch", + path: "SKILL.md", + detail: + "skill_versions.content (served by /raw) differs from SKILL.md in the stored tree", + }); + } + } + } + + return { + ok: findings.length === 0, + findings, + treeHashFromManifest, + treeHashFromBytes, + }; +} + +export interface TreeHashSweepRow { + versionId: string; + skillId: string; + version: number; + label: string; + storedTreeHash: string | null; + manifest: unknown; + storageBackend: string | null; + content: string | null; +} + +export interface TreeHashSweepQueryOptions { + latestOnly: boolean; + skillId: string | null; + limit: number | null; +} + +/** Column shape returned by buildSweepVersionQuery. */ +export interface TreeHashSweepVersionRecord { + version_id: string; + skill_id: string; + version: number; + content: string | null; + files: unknown; + tree_hash: string | null; + storage_backend: string | null; + public_author_slug: string | null; + public_slug: string | null; +} + +type SqlTemplateTag = ( + strings: TemplateStringsArray, + ...values: unknown[] +) => Q; + +/** + * The sweep's only query. SELECT-only on purpose: the sweep must not change data. + * `sql` is a tagged-template client such as neon(DATABASE_URL); fragments compose. + */ +export function buildSweepVersionQuery( + sql: SqlTemplateTag, + options: TreeHashSweepQueryOptions +): Q { + const skillFilter = options.skillId + ? sql`AND v.skill_id = ${options.skillId}::uuid` + : sql``; + const limitClause = options.limit ? sql`LIMIT ${options.limit}` : sql``; + + if (options.latestOnly) { + return sql` + SELECT * FROM ( + SELECT DISTINCT ON (v.skill_id) + v.id AS version_id, v.skill_id, v.version, v.content, v.files, + v.tree_hash, v.storage_backend, s.public_author_slug, s.public_slug + FROM skill_versions v + JOIN skills s ON s.id = v.skill_id + WHERE true ${skillFilter} + ORDER BY v.skill_id, v.version DESC + ) latest + ORDER BY latest.skill_id + ${limitClause} + `; + } + return sql` + SELECT + v.id AS version_id, v.skill_id, v.version, v.content, v.files, + v.tree_hash, v.storage_backend, s.public_author_slug, s.public_slug + FROM skill_versions v + JOIN skills s ON s.id = v.skill_id + WHERE true ${skillFilter} + ORDER BY v.skill_id, v.version + ${limitClause} + `; +} + +export function sweepRowFromVersionRecord( + record: TreeHashSweepVersionRecord +): TreeHashSweepRow { + const route = + record.public_author_slug && record.public_slug + ? `${record.public_author_slug}/${record.public_slug}` + : record.skill_id; + return { + versionId: record.version_id, + skillId: record.skill_id, + version: record.version, + label: `${route} v${record.version}`, + storedTreeHash: record.tree_hash, + manifest: record.files, + storageBackend: record.storage_backend, + content: record.content, + }; +} + +export interface TreeHashSweepEntry { + row: TreeHashSweepRow; + result: TreeHashAuditResult; +} + +export interface TreeHashSweepSummary { + checked: number; + ok: number; + withFindings: number; + findingsByCode: Partial>; + failures: TreeHashSweepEntry[]; +} + +/** + * Audit each row in order. loadFiles reads the stored bytes; a thrown error is + * reported as files_unavailable for that row and the sweep continues. + */ +export async function sweepTreeHashes( + rows: Iterable | AsyncIterable, + loadFiles: (row: TreeHashSweepRow) => Promise, + onResult?: (entry: TreeHashSweepEntry) => void +): Promise { + const summary: TreeHashSweepSummary = { + checked: 0, + ok: 0, + withFindings: 0, + findingsByCode: {}, + failures: [], + }; + + for await (const row of rows) { + let files: TreeFile[] | null = null; + let loadError: string | null = null; + try { + files = await loadFiles(row); + } catch (error) { + loadError = (error as Error)?.message ?? String(error); + } + + const result = auditSkillVersionTree({ + storedTreeHash: row.storedTreeHash, + manifest: row.manifest, + content: row.content, + files, + loadError, + }); + const entry = { row, result }; + summary.checked += 1; + if (result.ok) { + summary.ok += 1; + } else { + summary.withFindings += 1; + summary.failures.push(entry); + for (const finding of result.findings) { + summary.findingsByCode[finding.code] = + (summary.findingsByCode[finding.code] ?? 0) + 1; + } + } + onResult?.(entry); + } + + return summary; +} diff --git a/web/package.json b/web/package.json index 03fdb091..cfc3c359 100644 --- a/web/package.json +++ b/web/package.json @@ -13,6 +13,7 @@ "db:phase6-chain-identity": "tsx scripts/phase6-chain-identity-migration.ts", "db:walletless-buyers": "tsx scripts/walletless-buyer-migration.ts", "db:widen-protocol-version": "tsx scripts/widen-protocol-version-columns.ts", + "db:tree-hash-sweep": "tsx scripts/tree-hash-sweep.ts", "x402:bridge-poc": "tsx scripts/x402-bridge-poc.ts", "base:a1:ops": "tsx scripts/base-paid-report-e2e-smoke.ts", "stripe:ops": "tsx scripts/stripe-limited-preview-ops.ts", diff --git a/web/scripts/tree-hash-sweep.ts b/web/scripts/tree-hash-sweep.ts new file mode 100644 index 00000000..4ce1838b --- /dev/null +++ b/web/scripts/tree-hash-sweep.ts @@ -0,0 +1,247 @@ +/** + * Read-only tree-hash sweep (Phase 1 of .agents/plans/skill-evals-and-packages.plan.md). + * + * Recomputes the tree hash (agentvouch-tree-v1, docs/SKILL_TREE_HASH.md) of stored + * skill versions from their bytes, and compares it with skill_versions.tree_hash and + * the stored file manifest. It also checks that skill_versions.content, which + * /api/skills/{id}/raw serves for SKILL.md, equals SKILL.md in the stored tree. + * Run it before the CLI starts to stop installs on a hash mismatch. + * + * Read-only by design: + * - SELECT queries only, through a direct neon client. It never calls + * initializeDatabase(), because that runs schema DDL and backfills. + * - Blob reads only (get). No put or delete. + * - It does not call the public download routes, because those routes record + * download events and change public install counts. + * + * Usage (from web/): + * npx tsx --env-file=.env.local ./scripts/tree-hash-sweep.ts [options] + * DATABASE_URL=... BLOB_READ_WRITE_TOKEN=... \ + * npm run db:tree-hash-sweep --workspace @agentvouch/web -- [options] + * + * Options: + * --latest-only Check only the newest version of each skill (default: all versions). + * --skill Check one skill (DB UUID). + * --limit Check at most n versions. + * --json Write every result, with findings and recomputed hashes, to a file. + * --verbose Print one line per version, not only the failures. + * + * Env: + * DATABASE_URL Required. The target host is printed before any query. + * EXPECTED_DATABASE_HOST Optional. When set, the sweep stops unless it matches. + * BLOB_READ_WRITE_TOKEN Required to read blob-backed (multi-file) versions. + * + * Exit codes: 0 every version matches; 1 one or more versions have findings; + * 2 setup or usage error. + */ +import { writeFileSync } from "fs"; +import { neon } from "@neondatabase/serverless"; +import { buildArchiveForVersion, ingestTarArchive } from "@/lib/skillStorage"; +import { + buildSweepVersionQuery, + sweepRowFromVersionRecord, + sweepTreeHashes, + TREE_HASH_ALGORITHM, + type TreeFile, + type TreeHashSweepEntry, + type TreeHashSweepQueryOptions, + type TreeHashSweepRow, + type TreeHashSweepVersionRecord, +} from "@/lib/treeHashAudit"; + +type SweepOptions = TreeHashSweepQueryOptions & { + jsonPath: string | null; + verbose: boolean; +}; + +const UUID_PATTERN = + /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; + +class UsageError extends Error {} + +function parseArgs(argv: string[]): SweepOptions { + const options: SweepOptions = { + latestOnly: false, + skillId: null, + limit: null, + jsonPath: null, + verbose: false, + }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + const next = () => { + const value = argv[index + 1]; + if (value === undefined || value.startsWith("--")) { + throw new UsageError(`${arg} needs a value`); + } + index += 1; + return value; + }; + if (arg === "--latest-only") options.latestOnly = true; + else if (arg === "--verbose") options.verbose = true; + else if (arg === "--skill") { + const value = next(); + if (!UUID_PATTERN.test(value)) { + throw new UsageError("--skill must be a skill DB UUID"); + } + options.skillId = value.toLowerCase(); + } else if (arg === "--limit") { + const value = Number(next()); + if (!Number.isInteger(value) || value <= 0) { + throw new UsageError("--limit must be a positive integer"); + } + options.limit = value; + } else if (arg === "--json") options.jsonPath = next(); + else throw new UsageError(`Unknown option: ${arg}`); + } + return options; +} + +function getDatabase() { + const url = process.env.DATABASE_URL; + if (!url) { + throw new UsageError( + "DATABASE_URL is required. Point it at the intended Neon project (live agentvouch-postgres, not the legacy agent-reputation-oracle project)." + ); + } + const host = new URL(url).hostname; + const expected = process.env.EXPECTED_DATABASE_HOST?.trim(); + if (expected && expected !== host) { + throw new UsageError( + `EXPECTED_DATABASE_HOST (${expected}) does not match the DATABASE_URL host (${host}). Nothing was read.` + ); + } + return { db: neon(url), host }; +} + +/** Same sources the download routes use: inline content, or the blob tar by tree_hash. */ +async function loadStoredFiles(row: TreeHashSweepRow): Promise { + if (row.storageBackend !== "blob") { + if (row.content === null) { + throw new Error("Inline version has no content"); + } + return [{ path: "SKILL.md", bytes: Buffer.from(row.content, "utf8") }]; + } + if (!row.storedTreeHash) { + throw new Error( + "Blob-backed version has no tree_hash to locate its archive" + ); + } + const archive = await buildArchiveForVersion({ + content: row.content ?? "", + files: null, + tree_hash: row.storedTreeHash, + storage_backend: row.storageBackend, + }); + return ingestTarArchive(archive).map((file) => ({ + path: file.path, + bytes: Buffer.isBuffer(file.content) + ? file.content + : Buffer.from(file.content), + })); +} + +function describeEntry({ row, result }: TreeHashSweepEntry): string { + if (result.ok) return `ok ${row.label}`; + const codes = Array.from(new Set(result.findings.map((f) => f.code))); + return `FAIL ${row.label} (${row.skillId}): ${codes.join(", ")}`; +} + +async function main(): Promise { + let options: SweepOptions; + let database: ReturnType; + try { + options = parseArgs(process.argv.slice(2)); + database = getDatabase(); + } catch (error) { + if (error instanceof UsageError) { + console.error(error.message); + return 2; + } + throw error; + } + + console.log(`Tree-hash sweep (${TREE_HASH_ALGORITHM}), read-only`); + console.log(`Database host: ${database.host}`); + console.log( + `Scope: ${ + options.latestOnly ? "latest version of each skill" : "all versions" + }${options.skillId ? `, skill ${options.skillId}` : ""}${ + options.limit ? `, limit ${options.limit}` : "" + }` + ); + if (!process.env.BLOB_READ_WRITE_TOKEN) { + console.warn( + "BLOB_READ_WRITE_TOKEN is not set: blob-backed versions will report files_unavailable." + ); + } + + const records = (await buildSweepVersionQuery( + database.db, + options + )) as TreeHashSweepVersionRecord[]; + const rows = records.map(sweepRowFromVersionRecord); + console.log(`Versions to check: ${rows.length}`); + + const results: TreeHashSweepEntry[] = []; + const summary = await sweepTreeHashes(rows, loadStoredFiles, (entry) => { + results.push(entry); + if (options.verbose || !entry.result.ok) { + console.log(describeEntry(entry)); + } + }); + + console.log(""); + console.log( + `Checked ${summary.checked}: ${summary.ok} match, ${summary.withFindings} with findings.` + ); + for (const [code, count] of Object.entries(summary.findingsByCode)) { + console.log(` ${code}: ${count}`); + } + + if (options.jsonPath) { + writeFileSync( + options.jsonPath, + `${JSON.stringify( + { + algorithm: TREE_HASH_ALGORITHM, + database_host: database.host, + generated_at: new Date().toISOString(), + options, + summary: { + checked: summary.checked, + ok: summary.ok, + with_findings: summary.withFindings, + findings_by_code: summary.findingsByCode, + }, + results: results.map(({ row, result }) => ({ + skill_id: row.skillId, + version_id: row.versionId, + version: row.version, + label: row.label, + storage_backend: row.storageBackend, + stored_tree_hash: row.storedTreeHash, + tree_hash_from_manifest: result.treeHashFromManifest, + tree_hash_from_bytes: result.treeHashFromBytes, + ok: result.ok, + findings: result.findings, + })), + }, + null, + 2 + )}\n` + ); + console.log(`Wrote ${options.jsonPath}`); + } + + return summary.withFindings > 0 ? 1 : 0; +} + +main() + .then((code) => { + process.exitCode = code; + }) + .catch((error) => { + console.error(error); + process.exitCode = 2; + }); From 198841f15c6e5ba2dca36452c8abf605ffd32c46 Mon Sep 17 00:00:00 2001 From: andy Date: Sun, 25 Oct 2026 00:43:02 -0700 Subject: [PATCH 2/2] docs(plan): record owner decisions for skill evals and packages D1-D11 and D14 approved with the recommended defaults. D12 stays off. D13: no evaluation model spending yet, so Phase 2 uses mock runs only. The owner runs the production sweep and merges every PR. --- .../plans/skill-evals-and-packages.plan.md | 30 ++++++++++--------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/.agents/plans/skill-evals-and-packages.plan.md b/.agents/plans/skill-evals-and-packages.plan.md index 793e371f..62276231 100644 --- a/.agents/plans/skill-evals-and-packages.plan.md +++ b/.agents/plans/skill-evals-and-packages.plan.md @@ -471,20 +471,20 @@ Candidate packages (P3): | ID | Decision | Recommendation | Main alternative | Status | | --- | --- | --- | --- | --- | -| D1 | Can evidence grant `allow`? | No. Evidence can only lower trust or inform | Let strong evidence raise trust | Proposed (keeps the current rule) | -| D2 | Version identity | Existing `tree_hash` with a written spec, plus archive SHA-256 and upstream commit SHA | A new hash scheme | Proposed | -| D3 | Case format | `claude plugin eval` layout; import `evals/evals.json`; Harbor later | A custom AgentVouch format | Proposed | -| D4 | Where runs happen | GitHub Actions; stdlib Python orchestrator outside npm workspaces; separate run and sign jobs; never on fork PRs | Vercel Sandbox started from the web app (good later for on-demand runs) | Proposed | -| D5 | What P2 may run | First-party and allowlisted publishers only; community skills wait for P4 | Run everything from day one | Proposed | -| D6 | Report signing | in-toto Statement in DSSE, Ed25519 key in a GitHub environment, public keys in `.well-known`, hash-chained log | Sigstore keyless now (no key custody, but a new verification dependency) | Proposed | -| D7 | Holdouts | Private repo, public hash commitments, quarterly rotation, reveal retired cases | Public cases only | Proposed | -| D8 | Trust rollup | New signals are display-only until P5 | Feed `recommended_action` now | Proposed | -| D9 | Curation | AgentVouch is the only curator in v1; 2 to 4 skills per package; pinned; policy plus human review; no payments, no stake | Open curation | Proposed | -| D10 | Distribution order | CLI first, then the Claude Code marketplace URL, then a Git mirror | Git mirror first (reaches Codex and the skills CLI sooner, weaker pins) | Proposed | -| D11 | Licensing | Host only permissive licenses (existing gate plus repo-root and frontmatter fallback, deny wins); reference mode for all others | Host anything with attribution | Proposed | -| D12 | Revocation reach | Yank and delist; leave `forceRemoveDeletedPlugins` off until the owner decides | Turn it on at launch | Open (Q4) | -| D13 | Spend | Hard caps per run (`--max-cost-usd`) and per month (`eval_budget_counters`) | No cap | Open (Q5) | -| D14 | Chain | No on-chain change. On-chain anchoring later; the ERC-8004 Validation Registry on Base Sepolia is the main candidate | Anchor now | Proposed | +| D1 | Can evidence grant `allow`? | No. Evidence can only lower trust or inform | Let strong evidence raise trust | Approved 2026-10-25 | +| D2 | Version identity | Existing `tree_hash` with a written spec, plus archive SHA-256 and upstream commit SHA | A new hash scheme | Approved 2026-10-25 | +| D3 | Case format | `claude plugin eval` layout; import `evals/evals.json`; Harbor later | A custom AgentVouch format | Approved 2026-10-25 | +| D4 | Where runs happen | GitHub Actions; stdlib Python orchestrator outside npm workspaces; separate run and sign jobs; never on fork PRs | Vercel Sandbox started from the web app (good later for on-demand runs) | Approved 2026-10-25 | +| D5 | What P2 may run | First-party and allowlisted publishers only; community skills wait for P4 | Run everything from day one | Approved 2026-10-25 | +| D6 | Report signing | in-toto Statement in DSSE, Ed25519 key in a GitHub environment, public keys in `.well-known`, hash-chained log | Sigstore keyless now (no key custody, but a new verification dependency) | Approved 2026-10-25 | +| D7 | Holdouts | Private repo, public hash commitments, quarterly rotation, reveal retired cases | Public cases only | Approved 2026-10-25 | +| D8 | Trust rollup | New signals are display-only until P5 | Feed `recommended_action` now | Approved 2026-10-25 | +| D9 | Curation | AgentVouch is the only curator in v1; 2 to 4 skills per package; pinned; policy plus human review; no payments, no stake | Open curation | Approved 2026-10-25 | +| D10 | Distribution order | CLI first, then the Claude Code marketplace URL, then a Git mirror | Git mirror first (reaches Codex and the skills CLI sooner, weaker pins) | Approved 2026-10-25 | +| D11 | Licensing | Host only permissive licenses (existing gate plus repo-root and frontmatter fallback, deny wins); reference mode for all others | Host anything with attribution | Approved 2026-10-25 | +| D12 | Revocation reach | Yank and delist; leave `forceRemoveDeletedPlugins` off until the owner decides | Turn it on at launch | Open (Q4); stays off until the owner decides | +| D13 | Spend | Hard caps per run (`--max-cost-usd`) and per month (`eval_budget_counters`) | No cap | Decided 2026-10-25: no model spending yet; mock runs only | +| D14 | Chain | No on-chain change. On-chain anchoring later; the ERC-8004 Validation Registry on Base Sepolia is the main candidate | Anchor now | Approved 2026-10-25 | ## Phases @@ -631,4 +631,6 @@ Each phase is one or more small PRs off current `main`. P3 and later need their - 2026-10-24 (`p1-tree-hash-spec`, branch `feat/skill-tree-hash-spec`): Added `docs/SKILL_TREE_HASH.md`, 10 vectors plus 9 invalid paths in `packages/agentvouch-protocol/test-vectors/tree-hash.json`, the stdlib Python reference `tree_hash_reference.py` (it produced the expected values), and `web/__tests__/lib/treeHashVectors.test.ts`. Production `computeTreeHash` and the Python reference agree on every vector. Mutation check: locale-aware sorting, code point sorting, and NFC path normalization each fail the vectors. The CLI half of Done-when lands with `p1-cli-verify`, so this todo stays in progress. - 2026-10-24 (`p1-hash-sweep`): Added `web/lib/treeHashAudit.ts` (pure audit, the SELECT-only query builder, behavioral tests) and `web/scripts/tree-hash-sweep.ts` (`npm run db:tree-hash-sweep`). The sweep reads the database and Blob storage directly. It does not use the public download routes, because they record download events. It also checks that `skill_versions.content`, which `/raw` serves for `SKILL.md`, equals `SKILL.md` in the stored tree. It has not run on production yet: it needs `DATABASE_URL` and `BLOB_READ_WRITE_TOKEN` from the owner's environment. - 2026-10-24: New finding: upload does not reject a tree where one path is also a folder of another path (`docs` and `docs/a.md`). Such a tree cannot be written to disk. The sweep reports it as `path_conflict`. The fix moved into `p1-cli-verify`. +- 2026-10-25: PR 1 opened as #262 (`feat/skill-tree-hash-spec`). CI (`test`, `contracts`) and the Vercel preview build passed. +- 2026-10-25: Owner answers. The owner runs the production sweep on their Mac. The owner reviews and merges every PR. No evaluation model spending yet (D13): P2 code is built and tested with mock runs only, so the "one real run" part of `p2-eval-runner` waits for a budget. Recommended defaults approved for D1-D11 and D14. D12 (auto-uninstall) stays off. Still open: Q2 (v1 default is AgentVouch-only results), Q3, Q4, Q8. - 2026-10-24: New finding (not yet seen in data): `ingestTarArchive` enforces the upload limits (200 files, 5 MB), but mirror sync stores trees up to 512 files and 8 MB. `/raw?path=` and `/zip` read stored trees through it, so they would fail for a mirrored tree above the upload limits; `/archive` would not. The sweep reports such versions as `files_unavailable` with the limit message. If the production sweep finds any, fix the reader to accept the limits the tree was stored with.