Skip to content

[Benchmark] 计划:支持组件开关矩阵(hide-official / ptc / discovery 等) #296

Closed

Description

@Richeir

你想解决什么问题

背景

bench 现在只能回答"装/不装 dsh-ops",不能回答"装哪几个组件、开哪几个开关"。#88 和 #207 的核心争议正是 hide-official 这类开关:替换官方工具到底让模型更省,还是更容易把整会话拖进 READINESS_BLOCKED?没有开关矩阵,这些争议只能靠零散真机报告,无法沉淀成可比证据。

#262 之后报告可读性提升,但前提是报告里能看出"这次到底开了什么"。

现状(代码事实)

目标

  1. bench setup --switch <name>[=<value>](可重复)只写 candidate profile 自己的 patch 层,baseline 永不加 dsh-ops 行;名字→行的映射写成数据表,未知名字报错并列出可用名字。
    建议先支持:hide-official(replaceTools 列表或 all)、ptc、panel、discovery、shell。
  2. 幂等 + --dry-run + --force:照抄 pwshPath / permission mode 的语义;不认识的既有写入必须拒绝覆盖,不能静默改。
  3. 记录:run.json 增加 arms.cand.switches = { "hide-official": { ... }, ... }(run.schema.json 同步),profileHash 仍保留;报告头与 panel 显示开关集合。
  4. 可比性:compare 校验两份 run 的 candidate 开关集一致;baseline 的 switches 为空、candidate 非空是预期(这正是 A/B),但两份开关集不同的 candidate 不允许合并 Δ,必须显式报错。
  5. 就绪探测:setup 或首次 ab 之前,用 dsh --dump-config + 一次最小 headless 调用确认会话能起来;READINESS_BLOCKED 直接失败并指向 [BUG] 0.3.2 Windows 真机实测:启用 hide-official 后整会话 READINESS_BLOCKED(EXECUTOR_UNAVAILABLE)——48 个官方工具无等价物,stock 桌面环境确定性不可用(#178/#169) #207,不产出"全红"的 sweep。
  6. 测试:映射表、幂等、拒绝覆盖、结构化记录、开关不一致拒绝、就绪失败路径(注入假 dsh);README 增"组件开关"一节;CHANGELOG。

验收

不在本切片

  • 不自动搜索"最优开关组合"(组合爆炸,另议)。
  • 不改 hide-official 本身的行为。
  • 不把开关矩阵放进 CI 门禁。

关联

#262(报告可读性)、#292(对照数据动机)、#88(禁用官方工具开关)、#207(hide-official 就绪即死)、#205(bench 边界)。

现状与替代方案

N/A

建议方案(可选)

No response

是否愿意提 PR

愿意,并已能本地跑通门禁

Activity

  1. Haoming9527 commented on Oct 24, 2026

    @Haoming9527
    Collaborator

    Working on the component-switch slice on top of the model-condition and platform benchmark changes. The candidate profile alone will receive explicit, validated switches; existing conflicting values require --force. I will record the effective switch sets, refuse incompatible candidate comparisons, and fail setup before a sweep when the requested configuration cannot pass a bounded readiness probe. Default behavior without switch flags stays unchanged.

  2. Haoming9527 commented on Oct 24, 2026

    @Haoming9527
    Collaborator

    The candidate component-switch implementation is ready in #404 (8481d91), stacked on #402 to keep the review diff bounded. It supports repeatable switches, safe/idempotent candidate patching, explicit force/dry-run behavior, composed-config verification and a bounded readiness turn. Reports and sealed arm descriptors retain actual switch facts, and comparisons reject different candidate sets with exit 2 while allowing the baseline's expected empty set.

    All twenty benchmark suites passed locally. Ten dedicated checks cover dynamic/legacy config handling, the official host configuration merger, readiness failures, CLI rejection and the rendered panel. Current-head PR CI passed every executed platform and the Node floor: https://github.com/T-Auto/dsh-ops/actions/runs/40114967750.

    #207 remains open for Desktop validation; successful headless setup is not presented as that receipt. No model-based token savings or automatic search for an optimal switch combination is claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions