Repository navigation
feat(bench): route and separate model reasoning conditions - #402
Merged
Merged
Conversation
Haoming9527
marked this pull request as ready for review
October 24, 2026 17:05
This was referenced Oct 24, 2026
Haoming9527
dismissed
Thatgfsj’s stale review
October 25, 2026 04:36
The merge-base changed after approval.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #294
What was wrong
--modelonly labeled benchmark results; it did not change the route used by either measured arm. Reasoning effort was not a recorded comparison condition, so differently routed runs could be paired without a visible refusal.The change
Model and reasoning selections now produce a final
agent-default-modelpatch shared by both arms. The preflight checks the actual composed--dump-configroute before any measured session, archives that configuration, and records{ provider, model, reasoningEffort }in summaries and sealed evidence. The original LLM overlay is unchanged. No flags preserve existing routing and record an unknown condition; historical model labels are not treated as verified routing.Repeated
--model/--reasoning-effortflags run a Cartesian matrix in separate condition directories. Reports and the panel show conditions; cross-condition pairing fails with both conditions shown. Matrix reports keep separate sections and never combine deltas. An interrupt stops all remaining conditions.--providersupplies a route explicitly when it cannot be inferred from a literal overlay.This draft includes the existing report work in #381 and portability work in #399, preserving their history. Review/merge order: #381, then #399, then this PR. The #294-specific diff can be reviewed against
80f210e. It targets main so the repository's current pull-request workflow runs; current main662a388is already merged into this branch. No dependency PR has been merged by this change.The combined main diff is 960 implementation lines added/removed, including workflow, schema, task and example configuration changes, excluding tests and documentation. No dependency, package version, published tool schema or runtime prompt changed. Bilingual benchmark documentation and changelog are included. CI performs no model sweep.
Validation
The new matrix regression uses the official host's
renderConfigDumpto compose the real ordered patches, then exercises the session driver, sealed schema, CLI comparison and browser panel with a scripted child. It verifies the exact route and identical patches for both arms across four model/effort conditions, old-run unknown behavior, preflight refusal before any session, and separate matrix reports. No paid model calls were made.Behavioral red/green: restoring the previous session driver failed the actual route assertion (1 of 5 checks failed); removing the condition pairing guard failed the mixed-condition assertion (1 of 5 failed). Restoring both passed all 5. An added interruption regression failed with two condition sweeps instead of one (1 of 6 failed); sharing the interrupt state passed all 6. All 198 benchmark checks then passed after integrating portability and main.
The complete local native/full verification gate was not rerun for this benchmark-only change. The current-head pull_request CI run passed against
ddbc750: Linux, macOS, Windows, node-floor, tools-trial, workflow-lint, source-size andCI requiredall succeeded. The optionalbench-smokejob was skipped. This final prerequisite-history merge has the exact same tree as locally tested6fa6738(16fc2580b5f91b325e52511916d396ae2b83e324). The preceding run40113429295 was cancelled by the new push, not reported as a passing run.Environment
Windows x64, Node v24.19.0, local checkout with locked dependencies shared from an isolated installation. Main base
662a388; preserved benchmark ancestors #381 (f1dafcc) and #399 (80f210e).