Skip to content

Optimize skill token cost across all families (setup-family recipe) #1342

Description

@potiuk

Make every skill family leaner without changing what its skills do, using the recipe first applied to the setup family (#1343, PR #1353).

Why

Every skill costs tokens twice:

  • Always-on — description + when_to_use are loaded in every session for every installed skill, whether it runs or not. Budget: 200 tokens per skill.
  • Body — SKILL.md is loaded on every invocation. Budget: 5,000 tokens, pre-flight block excluded; PRINCIPLES.md P14's 500-line cap is the structural limit.

Both budgets come from optimize-skill, which is also the tool for doing this work.

Recipe (from the setup family)

Run /magpie-optimize-skill per skill and apply the passes in this order:

  1. Measure first. uv run --project tools/skill-token-count skill-token-count --write, plus a chars ÷ 4 estimate of the frontmatter. Record before/after for both budgets.
  2. Trim the always-on frontmatter of any skill over 200 tokens. Drop lists that argument-hint or the body already carries; keep the routing phrases a model needs to pick the skill. Highest value per token, because it is paid in every session.
  3. Split (only for bodies over budget, or rules that serve one sub-action). Move such rules into the sibling file that uses them, byte-for-byte, leaving a one-line pointer. Keep a rule in SKILL.md if many places cite it. Repoint every #anchor reference to what moved.
  4. Wording pass. One sentence per line (SemBr); cut sentences that restate a nearby table or section; plain verbs; no hedges. Keep bold only on real prohibitions.
  5. Fix stale pointers found on the way (links to renamed files, "above" that is below, wrong counts). List anything eval-coupled that you did not fix in the PR description.

Invariants — a PR that breaks one is not an optimization

  • Headings (#–####) and **Golden rule N — …** headlines stay byte-identical. They feed surface_hash (changing one tells every adopter their configuration is stale) and eval step_heading matching.
  • Frontmatter other than description / when_to_use, the generated pre-flight block, and every fenced code block stay byte-identical.
  • Every link target, command, flag, path, placeholder and condition survives. Check it mechanically: compare headings, fences, links and inline code spans between HEAD and the working tree, with whitespace normalised.
  • Eval-extracted sections (step_heading in tools/skill-evals/evals/<skill>/*/step-config.json): tighten only; every string that fixtures or expected.json rely on stays byte-identical.
  • The validator, prek run --all-files and lychee are green, and docs/mode-economics.md plus any family token figure in docs/setup/marketplace.md are regenerated.
  • Run the eval suites before and after. The harness launches the claude CLI, which fails inside the sandbox, so run it outside, or every case errors and nothing is graded.

What to expect

On setup, the family with no skill over either budget, the result was −2,709 body tokens across 7 skills (−8.5%) and the one frontmatter over budget cut from ~228 to ~167 tokens. The big win was setup/SKILL.md (−16%), where rules were moved as well as reworded; pure wording passes gave 4–9%. Families with skills far over budget (security, release-management, pr-management, issue) should gain much more from the split pass than setup did.

Sub-issues

One per family; each lists its skills with both budgets and marks the ones exceeded. Suggested priority: always-on first (it is paid by every session), then the largest bodies.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    capability:authoringSkills that author or maintain other skills: write-skill, optimize-skillenhancementNew feature or requestkind:perfToken / latency / API-call budget

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions