Skip to content

feat(bench): route and separate model reasoning conditions - #402

Merged
T-Auto merged 5 commits into
mainfrom
work/resolve-294-model-matrix
Oct 25, 2026
Merged

T-Auto merged 5 commits into
mainfrom
work/resolve-294-model-matrix

Conversation

@Haoming9527

@Haoming9527 Haoming9527 commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #294

What was wrong

--model only labeled benchmark results; it did not change the route used by either measured arm. Reasoning effort was not a recorded comparison condition, so differently routed runs could be paired without a visible refusal.

The change

Model and reasoning selections now produce a final agent-default-model patch shared by both arms. The preflight checks the actual composed --dump-config route before any measured session, archives that configuration, and records { provider, model, reasoningEffort } in summaries and sealed evidence. The original LLM overlay is unchanged. No flags preserve existing routing and record an unknown condition; historical model labels are not treated as verified routing.

Repeated --model / --reasoning-effort flags run a Cartesian matrix in separate condition directories. Reports and the panel show conditions; cross-condition pairing fails with both conditions shown. Matrix reports keep separate sections and never combine deltas. An interrupt stops all remaining conditions. --provider supplies a route explicitly when it cannot be inferred from a literal overlay.

This draft includes the existing report work in #381 and portability work in #399, preserving their history. Review/merge order: #381, then #399, then this PR. The #294-specific diff can be reviewed against 80f210e. It targets main so the repository's current pull-request workflow runs; current main 662a388 is already merged into this branch. No dependency PR has been merged by this change.

The combined main diff is 960 implementation lines added/removed, including workflow, schema, task and example configuration changes, excluding tests and documentation. No dependency, package version, published tool schema or runtime prompt changed. Bilingual benchmark documentation and changelog are included. CI performs no model sweep.

Validation

$ node test/run.mjs bench
dsh-ops test: 19 suite(s) passed in 61s
# 198 checks, 0 skipped, 0 failed

$ node scripts/check-file-size.mjs
check-file-size: 390 code file(s) checked, all at or under 1000 lines (longest: bench/setup/profile.mjs (927))

$ node scripts/check-workflows.mjs
check-workflows: all 78 checks passed

$ git diff --check
# exit 0

The new matrix regression uses the official host's renderConfigDump to compose the real ordered patches, then exercises the session driver, sealed schema, CLI comparison and browser panel with a scripted child. It verifies the exact route and identical patches for both arms across four model/effort conditions, old-run unknown behavior, preflight refusal before any session, and separate matrix reports. No paid model calls were made.

Behavioral red/green: restoring the previous session driver failed the actual route assertion (1 of 5 checks failed); removing the condition pairing guard failed the mixed-condition assertion (1 of 5 failed). Restoring both passed all 5. An added interruption regression failed with two condition sweeps instead of one (1 of 6 failed); sharing the interrupt state passed all 6. All 198 benchmark checks then passed after integrating portability and main.

The complete local native/full verification gate was not rerun for this benchmark-only change. The current-head pull_request CI run passed against ddbc750: Linux, macOS, Windows, node-floor, tools-trial, workflow-lint, source-size and CI required all succeeded. The optional bench-smoke job was skipped. This final prerequisite-history merge has the exact same tree as locally tested 6fa6738 (16fc2580b5f91b325e52511916d396ae2b83e324). The preceding run40113429295 was cancelled by the new push, not reported as a passing run.

Environment

Windows x64, Node v24.19.0, local checkout with locked dependencies shared from an isolated installation. Main base 662a388; preserved benchmark ancestors #381 (f1dafcc) and #399 (80f210e).

Thatgfsj
Thatgfsj previously approved these changes Oct 24, 2026

@Thatgfsj Thatgfsj left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审无误,批准合并

@Haoming9527
Haoming9527 dismissed Thatgfsj’s stale review October 25, 2026 04:36

The merge-base changed after approval.

@T-Auto
T-Auto merged commit 6626432 into main Oct 25, 2026
10 checks passed
@T-Auto
T-Auto deleted the work/resolve-294-model-matrix branch October 25, 2026 05:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Benchmark] 计划:支持多模型与多思考等级对比

3 participants