examples: add skill-creator description auth regression case - #15
Open
GBX-Max1220 wants to merge 2 commits into
Open
GBX-Max1220 wants to merge 2 commits into
GBX-Max1220 wants to merge 2 commits into
Conversation
Adds examples/skill-creator-description-auth. The fixture pins anthropics/skills at 7029232b9212482c0476da354b83364bd28fab2f (Skill tree 5c26c1472b98de33a48074ba5d3efe3a727b8e37), where the description optimizer builds an Anthropic SDK client and therefore requires ANTHROPIC_API_KEY. The upstream repair b0cbd3df1533b396d281a6886d5132f623393a9c (Skill tree 98a510d1cfef9f82c3b1eec200229ca68725e613, PR anthropics/skills#547) replaces the SDK client with a `claude -p` subprocess call. Both commits are fetched and verified by full SHA plus Skill tree and license blob 7a4a3ea2424c09fbe48d455aed1eaa94d9124835. The generated .test/ regression test has three clauses: structural detection of an ambient-credential SDK client (defect), structural requirement for the `claude` CLI invocation, and a behavioural run against a stub `claude` on PATH asserting prompt-over-stdin and <new_description> parsing. Clause C runs only when A and B hold. Evidence: baseline test fails (exit 1) reporting the SDK credential dependency; after applying repair.patch it passes (exit 0). Limitation (recorded in README and PROVENANCE): the defect's runtime auth error is not reproduced here because the anthropic package is not installed, so the failure is detected structurally rather than by triggering the SDK's error. No live API calls are made. Scope: examples/ only; no SkillHone core changes.
Clause C only checked stdin delivery and tag parsing, so deleting `-p` from the optimizer's `cmd = ["claude", "-p", ...]` left the test green: the flag that selects non-interactive one-shot mode was unasserted even though the clause B header claimed it was covered. Clause C now asserts the argv the stub `claude` received, and the TEST_SOURCE clause description is corrected (it labelled the behavioural clause as "Clause B"). Verified against a freshly generated fixture: the defect FAILs (exit 1), repair.patch PASSes (exit 0), and repair.patch with `-p` deleted FAILs (exit 1, reporting the argv it observed). README records that the assertion covers the stub's argv and not the real CLI's behaviour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a reproducible example for an authentication mismatch in
anthropics/skills' skill-creator description optimizer. At the pinned defect commit, the optimizer constructs an Anthropic SDK client that depends onANTHROPIC_API_KEY, even when the user has Claude Code authentication.What changed
examples/skill-creator-description-auth/README.mdandprepare_fixture.py.7029232b9212482c0476da354b83364bd28fab2f, skill tree, and license before generating the fixture..test/regression test inside the fixture. No upstream skill source is committed to SkillHone.b0cbd3df1533b396d281a6886d5132f623393a9cand its repair patch.Reproduction
From the SkillHone repository root:
python3 examples/skill-creator-description-auth/prepare_fixture.py /tmp/sc-auth-pr cd /tmp/sc-auth-pr python3 .test/test_description_optimizer_auth_contract.pyThe defect baseline exits 1. Structural checks identify SDK client construction with ambient credentials and the absence of a
claudeCLI call. Applying the generatedrepair.patchand rerunning the test gives exit 0; all three clauses pass.Validation and results
pnpm test: 31 passed, 0 failed.python3 skills/skillhone/scripts/quality/static_check.py skills/skillhone: passed.python3 skills/skillhone/scripts/quality/static_check.py skills/skillhone-auto-optimization: passed.gitleaks dir . --no-banner --redact: no leaks found.gitleaks git . --no-banner --redact: no leaks found.git diff --check: passed.Effect and limits
The baseline test detects the credential dependency structurally; it does not reproduce a live authentication error. The repaired optimizer is exercised with a stub
claudeexecutable to check prompt delivery through stdin and<new_description>parsing. No live API or model call is made. This PR changes only the two files underexamples/skill-creator-description-auth/; it does not change SkillHone core.Review checklist
git diff --check.gitleaksor an equivalent credential scan.