Skip to content

Re-tune the skill pack for Claude Fable 5.1 (prompt-fitness 59 → 95) - #1

Merged
Jibbscript merged 12 commits into
mastafrom
improve/fable_5_1_prompt_fitness
Sep 16, 2026
Merged

Jibbscript merged 12 commits into
mastafrom
improve/fable_5_1_prompt_fitness

Conversation

@Jibbscript

@Jibbscript Jibbscript commented Sep 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

Re-tunes the 12-skill pack for Claude Fable 5.1. The skills were written in the Opus 4.5 era as step scripts with copy-paste checklists; Fable 5.1 guidance says prompts written that way are too prescriptive and reduce output quality. This branch replaces choreography with goals, constraints, and reasons, and adds the Fable-era capabilities the pack never asked for.

Every change was produced by a tournament loop: each hypothesis was measured before execution, reviewed by an architect and a critic, executed in an isolated worktree, and re-benchmarked after merge. An independent verifier read the final files end to end.

Benchmark (eval/fable_prompt_bench.py, deterministic, stdlib-only, sealed during the loop): 59.37 → 95.25.

Dimension Before After
cruft (pressure language, scaffolds, fossils) 99.7 100
de_prescription (Step-N scripts, checklists) 53.3 100
context_richness (reasons, audience, prose) 4.2 ~70
fable_utilization (delegation, grounding, boundaries, judgment) 3.1 100
trigger_quality (frontmatter descriptions) 100 100

What changed in every skill

  • Fable capabilities — a ## Working <domain> section per skill, in that skill's own vocabulary: fan out independent research to parallel sub-agents, label every figure sourced / estimated / unverified, calibrate depth, name the downstream consumer, invite non-obvious judgment, lead the summary with the outcome, self-check before hand-off.
  • De-prescription — ### Step N scripts and Workflow Checklists replaced by ## Outcome, ## What <artifact> Must Cover, ## How to Work, ## Constraints. Every scale, field list, source list, query template, edge case, and calibration note is preserved; numbered order survives only where it genuinely matters.
  • Context — ## Who reads <artifact> and ## What a strong <artifact> looks like, with readership accurate to the producer/consumer table in contracts/; behavioral bullet walls reflowed into prose carrying the reason.
  • Structure — numeric scales rendered as tables, literal search-query templates fenced.
  • Orchestrator — tells the model the user is not watching and reversible steps proceed, asks for outcome-first final messages that re-ground a reader who saw none of the work, and ends the run either with all eleven artifacts or an explicit statement of what was skipped and why.
  • Docs — docs/CONTRIBUTING.md template and every AGENTS.md section-order line updated to match; four residual cruft lines cleared in the baseline runner prompt and three reference files.

Not changed

Frontmatter (names, descriptions, allowed-tools), contracts/, eval/fixtures/, LICENSE. The eval runners still list all five fixtures.

Test plan

  • python3 eval/fable_prompt_bench.py → 95.25, zero integrity failures, zero cross-skill boilerplate
  • Frontmatter diff vs masta empty; sealed paths byte-identical
  • python3 eval/runners/run_skillpack.py --list prints five fixtures
  • Independent verifier read risk-assessment, gtm-channels, and the orchestrator end to end (single voice, no contradictions, no lost domain knowledge)
  • Real-model check: fixture t1-energy-audit run three ways (new skills, old masta skills, no-skills baseline), each with live web research, graded by three rubric judges per run plus blind pairwise comparison in both orderings — see below
  • CI: prompt-fitness workflow added and passing on this PR

Real-model A/B on fixture t1 (Home Energy Leak Mapper)

Run Judge mean (0-100) Judge range Contract violations
new skills 88.2 85.5–92.0 5–10
old masta skills 87.7 81.0–91.5 5–8
no skills 79.5 77.5–82.0 8–26

Blind pairwise, position-corrected: new beats old in both orderings (slight, then clear) on usefulness and voice; correctness and citations are a wash. The two pack-vs-baseline pairs flipped with position and are not robust. Both pack runs beat the baseline mainly on contract compliance.

What the new pack did that the old one did not: corrected the fixture's false premise (no iPhone has a thermal sensor) and carried it through every artifact; wrote falsifiable next steps with kill thresholds; reported the unadjusted composite alongside the adjusted range that would flip the verdict. What the old pack did better: fieldwork depth (resolved the iOS UVC question, found one more competitor, surfaced California's referral-fee restriction).

Follow-ups surfaced by the judges, not addressed in this PR: both packs made the same derived-share arithmetic error in market sizing (recompute shares from the cited buckets); both missed the most direct competitor (search funded startups in the re-premised space); artifacts run 3–6× the reference length; the repo's Python scorer only measures artifact completion and cannot separate these runs.

The full report is eval/results/t1_ab_report.md on the machine that ran it (run output is gitignored by design).

🤖 Generated with Claude Code

Jibbscript and others added 12 commits September 15, 2026 22:18
Deterministic, stdlib-only scorer for the skill pack's prompt surface,
built from Anthropic's prompt-audit patterns and the Fable 5.1 migration
guidance. Baseline 59.37/100. Ignore self-improve worktrees.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…carrying Fable 5.1 capabilities (parallel sub-agents, sourced/estimated/unverified marking, depth, boundaries, downstream consumer, judgment, outcome-first, self-check) lifts fable_utilization from 3 to 100

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…er skill in its own domain vocabulary (score: 59.37 → 76.80)

Adding a short, purely additive Fable-capabilities passage to each of the 12 SKILL.md files - written in that skill's own domain vocabulary and covering paralle

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rite (18db659) onto the fable_capabilities winner and reconcile the merged Outcome/How to Work/Constraints sections with each skill Working section so every fact is stated once, lifting de_prescription 54.76 to 100 and primary 76.80 to 89.62

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…sts with outcome, coverage, and constraint prose, reconciled with the Working sections (score: 76.80 → 89.62)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tions from 7aed9d6 into all twelve skills and close the carrier-phrase and restatement seams (primary 89.62 -> 93.53, ctx 37.90 -> 62.36, dups 0)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…sections into every skill and close the carrier-phrase seams (score: 89.62 → 93.53)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… remaining chat-script sections

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…at-script sections deprescribed (score: 93.53 → 93.95)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…emplates as fenced blocks (prompt-audit 1c: structure for reference data)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…baseline runner prompt and three reference files (prompt-audit 1a/1d)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Runs eval/fable_prompt_bench.py on the PR head and the base branch; fails on
any integrity failure, a primary below 85, or a regression of more than 1.0
against base. Smoke-tests the eval runners. Writes the per-skill table to the
job summary.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Jibbscript
Jibbscript merged commit a0a5b22 into masta Sep 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant