Repository navigation
Re-tune the skill pack for Claude Fable 5.1 (prompt-fitness 59 → 95) - #1
Merged
Merged
Conversation
Deterministic, stdlib-only scorer for the skill pack's prompt surface, built from Anthropic's prompt-audit patterns and the Fable 5.1 migration guidance. Baseline 59.37/100. Ignore self-improve worktrees. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…carrying Fable 5.1 capabilities (parallel sub-agents, sourced/estimated/unverified marking, depth, boundaries, downstream consumer, judgment, outcome-first, self-check) lifts fable_utilization from 3 to 100 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…er skill in its own domain vocabulary (score: 59.37 → 76.80) Adding a short, purely additive Fable-capabilities passage to each of the 12 SKILL.md files - written in that skill's own domain vocabulary and covering paralle Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rite (18db659) onto the fable_capabilities winner and reconcile the merged Outcome/How to Work/Constraints sections with each skill Working section so every fact is stated once, lifting de_prescription 54.76 to 100 and primary 76.80 to 89.62 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…sts with outcome, coverage, and constraint prose, reconciled with the Working sections (score: 76.80 → 89.62) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tions from 7aed9d6 into all twelve skills and close the carrier-phrase and restatement seams (primary 89.62 -> 93.53, ctx 37.90 -> 62.36, dups 0) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…sections into every skill and close the carrier-phrase seams (score: 89.62 → 93.53) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… remaining chat-script sections Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…at-script sections deprescribed (score: 93.53 → 93.95) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…emplates as fenced blocks (prompt-audit 1c: structure for reference data) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…baseline runner prompt and three reference files (prompt-audit 1a/1d) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Runs eval/fable_prompt_bench.py on the PR head and the base branch; fails on any integrity failure, a primary below 85, or a regression of more than 1.0 against base. Smoke-tests the eval runners. Writes the per-skill table to the job summary. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Re-tunes the 12-skill pack for Claude Fable 5.1. The skills were written in the Opus 4.5 era as step scripts with copy-paste checklists; Fable 5.1 guidance says prompts written that way are too prescriptive and reduce output quality. This branch replaces choreography with goals, constraints, and reasons, and adds the Fable-era capabilities the pack never asked for.
Every change was produced by a tournament loop: each hypothesis was measured before execution, reviewed by an architect and a critic, executed in an isolated worktree, and re-benchmarked after merge. An independent verifier read the final files end to end.
Benchmark (
eval/fable_prompt_bench.py, deterministic, stdlib-only, sealed during the loop): 59.37 → 95.25.What changed in every skill
## Working <domain>section per skill, in that skill's own vocabulary: fan out independent research to parallel sub-agents, label every figure sourced / estimated / unverified, calibrate depth, name the downstream consumer, invite non-obvious judgment, lead the summary with the outcome, self-check before hand-off.### Step Nscripts and Workflow Checklists replaced by## Outcome,## What <artifact> Must Cover,## How to Work,## Constraints. Every scale, field list, source list, query template, edge case, and calibration note is preserved; numbered order survives only where it genuinely matters.## Who reads <artifact>and## What a strong <artifact> looks like, with readership accurate to the producer/consumer table incontracts/; behavioral bullet walls reflowed into prose carrying the reason.docs/CONTRIBUTING.mdtemplate and everyAGENTS.mdsection-order line updated to match; four residual cruft lines cleared in the baseline runner prompt and three reference files.Not changed
Frontmatter (names, descriptions,
allowed-tools),contracts/,eval/fixtures/,LICENSE. The eval runners still list all five fixtures.Test plan
python3 eval/fable_prompt_bench.py→ 95.25, zero integrity failures, zero cross-skill boilerplatemastaempty; sealed paths byte-identicalpython3 eval/runners/run_skillpack.py --listprints five fixturest1-energy-auditrun three ways (new skills, oldmastaskills, no-skills baseline), each with live web research, graded by three rubric judges per run plus blind pairwise comparison in both orderings — see belowprompt-fitnessworkflow added and passing on this PRReal-model A/B on fixture t1 (Home Energy Leak Mapper)
mastaskillsBlind pairwise, position-corrected: new beats old in both orderings (slight, then clear) on usefulness and voice; correctness and citations are a wash. The two pack-vs-baseline pairs flipped with position and are not robust. Both pack runs beat the baseline mainly on contract compliance.
What the new pack did that the old one did not: corrected the fixture's false premise (no iPhone has a thermal sensor) and carried it through every artifact; wrote falsifiable next steps with kill thresholds; reported the unadjusted composite alongside the adjusted range that would flip the verdict. What the old pack did better: fieldwork depth (resolved the iOS UVC question, found one more competitor, surfaced California's referral-fee restriction).
Follow-ups surfaced by the judges, not addressed in this PR: both packs made the same derived-share arithmetic error in market sizing (recompute shares from the cited buckets); both missed the most direct competitor (search funded startups in the re-premised space); artifacts run 3–6× the reference length; the repo's Python scorer only measures artifact completion and cannot separate these runs.
The full report is
eval/results/t1_ab_report.mdon the machine that ran it (run output is gitignored by design).🤖 Generated with Claude Code