Skip to content

Update front-page evaluation to the October 20 Sonnet 5.5 A/B - #257

Merged
m1rl0k merged 1 commit into
testfrom
site/oct20-sonnet55-numbers
Oct 20, 2026
Merged

m1rl0k merged 1 commit into
testfrom
site/oct20-sonnet55-numbers

Conversation

@m1rl0k

@m1rl0k m1rl0k commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

Replaces the October 5 Sonnet 4.5 figures with the October 20 Claude Sonnet 5.5 paired A/B (80 runs, 40 tasks x 2 seeds, same two repositories):

  • Tool calls per task: 14.2 -> 10.4 (-26%, p < 0.001)
  • Agent turns per task: 15.2 -> 11.4 (-25%, p < 0.001)
  • Model cost per task: $0.118 -> $0.094 (-20%, p = 0.02)
  • Accuracy held level (60.5% vs 61.3%); tokens -18% not significant, so not claimed

Task-clustered bootstrap CIs, task-level sign-flip permutation p-values. The earlier Sonnet 4.5 accuracy result is kept in the details text.

🤖 Generated with Claude Code

@m1rl0k
m1rl0k merged commit 7e39b34 into test Oct 20, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant