Skip to content

Close the three t1 A/B defects: derived shares, ladder monotonicity, funded-startup search - #2

Merged
Jibbscript merged 1 commit into
mastafrom
improve/fable_5_1_prompt_fitness
Sep 16, 2026
Merged

Jibbscript merged 1 commit into
mastafrom
improve/fable_5_1_prompt_fitness

Conversation

@Jibbscript

Copy link
Copy Markdown
Owner

Summary

The real-model A/B on eval/fixtures/t1-energy-audit (eval/results/t1_ab_report.md) found two arithmetic defects and one research miss the skills should have prevented. Each fix is a short reasoned paragraph in the skill's existing Fable 5.1 register (no CAPS, no checklists, no step scripts); frontmatter, contracts/, eval/, and LICENSE are untouched.

  • market-sizing — the model built "62% of homes pre-2000" from NAHB buckets by mislabelling a 2000–2009 bucket (correct figure ~72%). New paragraph in Working the Numbers: a derived share is recomputed in the artifact from the cited buckets, each boundary quoted as the source labels it, with the addition written out; the closing self-check re-adds any bucket-derived share from the quoted boundaries.
  • pricing-wtp — the generated value_ladder ran $0, $29, $79, $60, and ltv_cac_ratio was 3.6 while its own inline math gave 3.8. The ladder is checked for increasing price and increasing value before it is written (a rung that breaks the order is usually a different product wearing a tier label), and LTV:CAC is arithmetic over the file's own fields, stated once in ltv_cac_ratio with its formula beside it; the closing self-check covers both.
  • competitive-landscape — both pack runs missed HomeBoost, a funded direct competitor one TechCrunch search away. One search per landscape now goes to funded startups in the space the brief's solution hypothesis actually describes, plus two query templates.

Benchmark

python3 eval/fable_prompt_bench.py -v: primary 95.25 → 95.37, zero integrity failures, boilerplate_dups 0. Per-skill: competitive-landscape 93.7 → 94.6, market-sizing 96.6 → 96.7, pricing-wtp 92.2 → 93.0.

🤖 Generated with Claude Code

… 95.25 → 95.37)

The real-model A/B on eval/fixtures/t1-energy-audit (eval/results/t1_ab_report.md)
found two arithmetic defects and one research miss that the skills should have
prevented. Each fix is a short reasoned paragraph in the skill's existing register.

market-sizing: a derived share (the "62% pre-2000" built from NAHB buckets, one of
them mislabelled by a decade; the correct figure is ~72%) is recomputed in the
artifact from the cited buckets, with the boundaries quoted as the source labels
them and the sum written out; the closing self-check re-adds any such share from
the quoted boundaries.

pricing-wtp: the value ladder is checked for increasing price and increasing
value before it is written (the generated ladder ran $0, $29, $79, $60), and
LTV:CAC is arithmetic over the file's own fields, stated once in ltv_cac_ratio
with its formula beside it (the generated file reported 3.6 while its inline
math gave 3.8); the closing self-check covers both.

competitive-landscape: one search per landscape goes to funded startups in the
space the idea brief's solution hypothesis actually describes, plus two query
templates, because both pack runs missed HomeBoost, a funded direct competitor
one TechCrunch search away.

Benchmark: python3 eval/fable_prompt_bench.py primary 95.25 → 95.37, zero
integrity failures, boilerplate_dups 0. Frontmatter, contracts/, eval/, and
LICENSE untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Jibbscript
Jibbscript merged commit 74d22ac into masta Sep 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant