Skip to content

Publish an agent compatibility matrix backed by adapter contract tests - #132

Merged
richardmhope merged 4 commits into
mainfrom
claude/codebase-review-o7y3i9
Sep 24, 2026
Merged

richardmhope merged 4 commits into
mainfrom
claude/codebase-review-o7y3i9

Conversation

@richardmhope

Copy link
Copy Markdown
Collaborator

What & why

Adds docs/compatibility.md, a public compatibility matrix linked from the README and docs/adapters.md. It covers all eight adapters: the five native ones plus the copilot-prompt, cursor-rule and kiro-steering legacy adapters. For each adapter it gives:

  • a status: tested, supported or experimental, each defined on the page. When the only evidence is a related vendor package, the status is experimental, so both Cursor adapters are experimental;
  • project and global paths, and the environment variables that move them;
  • how the skill is invoked in that agent;
  • the minimum agent version;
  • the date, agent version and vendor sources it was checked against (2026-09-23).

Anything a primary source doesn't confirm is marked unverified. The page also explains how the project responds when a vendor deprecates or moves a location or format, including urgent releases.

CI now pins each adapter's contract. tests/fixtures/adapter-contracts/ holds a synthetic skill, a contracts.json, and the exact rendered file every adapter should produce, stamp included. contracts.json records paths, scope-error text (for install and for other commands), the expected frontmatter, and the behaviour with each config-directory variable set to an absolute path, empty, or relative. tests/test_adapter_contracts.py:

  • installs with every adapter into temp dirs and compares byte for byte. This is Windows-safe: UTF-8 reads, POSIX-relative paths, fixtures marked -text, no symlinks;
  • requires the fixture directory to hold exactly the contract's files;
  • checks each matrix row's paths, "not supported", "Moved by" and scope-error text against the contracts. Status, minimum versions, dates and notes are maintained by hand, and the page says so;
  • requires the matrix's <!-- adapter-contract: sha256:… --> line to match a digest of the contract files. contracts.json is hashed as canonical JSON without its _about notes;
  • requires CHANGELOG.md to mention the digest prefix, either under [Unreleased] or, once a release is cut, in the newest dated section. This is tested against prepare_release.cut_changelog.

Unsupported scopes now fail with an actionable message. install --agent cursor-rule --scope global says "Use --scope project, or --agent cursor (Agent Skills), which supports --scope global". status, uninstall and update name only --scope project, because the native adapter would act on different files.

Review. An independent review raised eight findings, all fixed here:

  • the scope hint now depends on the command;
  • the fixture file set is checked exactly;
  • the matrix is checked cell by cell;
  • the CHANGELOG rule survives a release cut;
  • the Cursor adapters are marked experimental;
  • the cursor-rule invocation caveat is added;
  • the claim about Codex $name resolution is hedged as unverified;
  • _about is excluded from the digest.

Closes #79

Tracking: #94

Type of change

  • Bug fix
  • New feature
  • New or updated skill
  • Docs only
  • Refactor / internal

Checklist

  • Ran uv run --extra dev ruff check . && uv run --extra dev ruff format --check . && uv run --extra dev mypy && uv run --extra dev pytest (817 passed, also with -W error::EncodingWarning and with --resolution lowest-direct)
  • Added or updated tests
  • Updated docs where relevant (README, docs/adapters.md, new docs/compatibility.md, CLAUDE.md)
  • Added a CHANGELOG.md entry under ## [Unreleased]
  • For a skill change: bumped that skill's version in meta.yaml (no skill content changed)

🤖 Generated with Claude Code

https://claude.ai/code/session_01HtiCGzpikMrkDYBkfQG5CX


Generated by Claude Code

Add docs/compatibility.md, linked from the README and docs/adapters.md. It
covers every adapter: the five native ones and the copilot-prompt,
cursor-rule and kiro-steering legacy adapters. Each row gives a status
(tested, supported or experimental, defined on the page), the project and
global paths, the environment variables that move them, how the skill is
invoked, the minimum agent version, and the date, agent version and vendor
sources it was checked against. The research behind it was done on
2026-09-23, and whatever it couldn't confirm is marked unverified. The page
also sets out how skilldeck responds when a vendor deprecates or moves a
skill location or format.

tests/fixtures/adapter-contracts/ pins each adapter's exact rendered file
(stamp included) for one synthetic skill, its project and global paths, its
--scope global error if it is project-only, and its behaviour with each
config-directory variable set to an absolute path, left empty or set to a
relative path. tests/test_adapter_contracts.py installs the skill with every
adapter into temporary directories and compares the results byte for byte.
The fixtures are marked -text in .gitattributes so Windows checkouts keep
them exact. The matrix embeds a sha256 digest of the fixtures, and the
changelog must mention its first 12 characters, so a format change fails CI
until the matrix and changelog are updated.

The error for an unsupported scope now names what works instead. For
example, "Use --scope project, or --agent cursor (Agent Skills), which
supports --scope global".

Closes #79.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HtiCGzpikMrkDYBkfQG5CX
- Only install's unsupported-scope error suggests the native adapter
  (--agent cursor). For status, uninstall and update, that adapter would
  act on different files, so those commands name only --scope project.
  Adapter.install checks the scope first, so direct API use gets the same
  message.
- The contract fixture directory must hold exactly the contract's files.
  The digest covers only those files, and covers contracts.json as
  canonical JSON without its _about notes.
- The matrix test now checks each row cell by cell:
  - project and global paths, with "not supported" and "n/a" for
    project-only adapters
  - "Moved by" names exactly the variables that move the global install,
    each with its target, and mentions the ones that don't
  - both quoted scope errors match the real messages
  The docs and CLAUDE.md say what is enforced and what is kept by hand.
- The CHANGELOG digest mention must be in [Unreleased] or the newest dated
  section, so the check still holds right after prepare_release cuts a
  release. Tested with synthetic changelogs and with the real
  cut_changelog.
- Cursor and cursor-rule are experimental: their only evidence is
  @cursor/sdk. Experimental is now defined to cover SDK-only evidence.
  Claims about the Cursor app and CLI, including env overrides, are marked
  unverified, and cursor-rule's invocation gets the same caveat.
- Codex's $name resolution is hedged: its two selection paths differ, and
  which surface uses which is unverified. docs/adapters.md is hedged too.

Part of #79.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HtiCGzpikMrkDYBkfQG5CX
# Conflicts:
#	CHANGELOG.md
#	CLAUDE.md
#	src/skilldeck/cli.py
@richardmhope
richardmhope merged commit 5dc64d6 into main Sep 24, 2026
17 checks passed
@richardmhope
richardmhope deleted the claude/codebase-review-o7y3i9 branch September 24, 2026 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Publish an automated agent compatibility matrix

2 participants