A bench that can embarrass its own project: 4 agents, 5 tasks, measured - #97
Merged
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Real third-party comparisons, as asked: same tasks, same fixture, measured, raw transcripts committed.
bench/run_bench.pyreproduces everything.Setup. Five deterministic tasks over a synthetic incident scene (three investigation tasks with planted facts, two mutation tasks pairing a goal with an explicit prohibition and a temptation). Graded on planted-fact checks and file hashes, not vibes. Agents:
claude-code(headless),opencode(same local qwen3:8b as GraphARC for a same-model pair),grapharc-qwen,grapharc-claude. Codex is installed and authenticated but its ChatGPT account tier rejects every exec model — skipped with the raw error recorded; pi is not installed. Both skips are inresults.json, not swept away.Pilot (n=1 per cell):
The numbers are not flattering and are published anyway: claude-code sweeps (it also runs a far stronger model than the local rows); the same-model pair (grapharc-qwen 1/5 vs opencode-qwen 0/5) supports no winner at this n; and the bench's first outing caught a real GraphARC defect — the delegated claude-cli path fails with an empty error and still reports
goal_met(#96).n=1 is a pilot, labeled as such in bench/README.md.
--repeat 3is the reproduce command for anything worth quoting.🤖 Generated with Claude Code