Skip to content

Integrate C1 and evidence-driven benchmark candidate selection - #8255

Closed
huangruiteng wants to merge 155 commits into
codex/edgebench-r35-signed-candidate-20261107from
codex/edgebench-r36-aggressive-20261107
Closed

huangruiteng wants to merge 155 commits into
codex/edgebench-r35-signed-candidate-20261107from
codex/edgebench-r36-aggressive-20261107

Conversation

@huangruiteng

@huangruiteng huangruiteng commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Long-running benchmark workers should spend fewer interactions recovering control context and use observed candidate outcomes when choosing their next experiment. This research integration combines C1 with candidate-selection guidance and optional process-group observations, including the real native benchmark entry paths.

This PR targets the preceding research integration branch. It does not promote this treatment to the default product or claim a measured task-score gain.

Changes

Evidence and limits

The complete native checkpoint/context regression selection passed 96 cases, with 11 platform skips. The candidate-selection checks exercised actual SQLite work and periodic projections as well as typed measurement scopes. The resource producer was independently checked on Linux against phase-shifted allocation, output-file failure and feature-off cases; the real wrapper installer was checked against missing and shadowed producers. Component PRs retain their exact-head reviews and required local gates.

A paired deterministic control journey reduced normal commands from four to three and model-visible bytes by about 26.7%. Including refusal and replay reduced bytes by about 17.6%; replay itself grew. Single-sample tool latency regressed, so these measurements do not establish lower total latency, billed token cost or better task outcomes. Host-internal capture transport savings are reported separately from model-visible savings.

Natural model adoption, sustained score growth and full cost remain unqualified. New single-treatment runs can test operational adoption and provide diagnostic comparisons with historical runs; they do not form a new matched four-arm experiment. Native account routing remains unchanged.

The combined branch is a research artifact. The upstream C1 default-retirement and broader natural-provider qualification gates remain open; component merge-readiness does not settle those gates.

Main delivery

The benchmark work is tracked in independent main-target PRs. The comparison finding is tracked with the Yardstick work, and the C1 composition is retained as a research candidate at this integration head. Capture reuse #8259 · Yardstick #8261 · Probe and memory evidence #8263 · C1 rollback #8265

loopx-agent and others added 30 commits November 5, 2026 05:46
…h main

Retain the complete checked decision across production callers, reuse main Progressive/Delta presentation and keep late transport options in derived views. Preserve original capture, numeric identity, failure and explicit-off semantics.

Co-authored-by: aihop <4408344+hhyykk@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Cover checked source fidelity, real File/SQLite caller settlement, native transport recovery, numeric observations and source-entrypoint parity. Retain independent semantic oracles and historical failure boundaries.

Co-authored-by: aihop <4408344+hhyykk@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…on limits

Keep paired RFC checkpoints, production caller contracts and historical negative cost evidence. Integration and retry results do not establish original-workload net benefit or installed delivery.

Co-authored-by: aihop <4408344+hhyykk@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
(cherry picked from commit dcc1d50)
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…ources

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…er-20261103T062012Z

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

# Conflicts:
#	docs/architecture/rfcs/controller-executor-unified-contract-v0.md
#	docs/architecture/rfcs/controller-executor-unified-contract-v0.zh-CN.md
#	loopx/cli_commands/quota_capture.py
#	loopx/control_plane/quota/turn_envelope.ts
#	loopx/control_plane/turn_driver/codex_execution_preparation.py
#	tests/control_plane/test_quota_execution_preparation.py
#	tests/control_plane/test_quota_settlement_cli.py
#	tests/control_plane/test_refresh_settle_turn.py
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent and others added 25 commits November 7, 2026 08:52
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…#8245)

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
feat(explore): ground work choices in paired goal outcomes
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Reuse the committed quota capture in native benchmark preflight
…ration

Observe aggregate memory in local work probes
Opt in to local work probes from SForge heartbeat

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent, gpt-6.1-sol, OpenAI, runtime_reported, reasoning_effort=max

[P2] 先按 Todo 筛选量测,再形成比较。 另一个已认领的 Todo 在同一 node 和量测范围写入 9 分后,原 Todo 仍保留原 finding 与 next_action,但 search_comparison 和紧凑 comparison_cue 消失。 按 Goal、Agent、Todo 和 node 过滤原 findings 后,复用既有量测聚合器计算比较。全局量测列表继续沿原路径。 聚合输入 · 当前比较

动机 / Motivation

实验操作者要接续同一次 benchmark,读取已保存的准入,并根据本任务的候选结果选择下一步。 原 worker 启动时传送完整 guard,候选比较主要展示量测。当前集成让 native reader 复用原捕获,并为当前 Todo 提供有作用域的比较提示。 同一原捕获能进入 native 工具链,正常情况下也能读到两候选的比较提示。另一个 Todo 更新同一研究问题后,本 Todo 的比较会消失。 本 PR 面向 r35 研究分支的 r36 集成。实际模型采用、同等预算下得分和完整成本,仍需独立验收。

改动思路 / Approach

继续用原捕获、量测与 provider owner。在现有量测投影中先选定同一 Todo 的发现,再计算候选比较,保留独立的全局结果视图。 本 PR 面向 r35 研究分支的 r36 集成。实际模型采用、同等预算下得分和完整成本,仍需独立验收。 C1 保留现有 source、reader 和结算身份。Host 在启动阶段校验指向原文件的摘要,模型读取仍沿原 transport。候选反馈沿现有 Explore 结果日志,PSS 内存估计沿已有本地 probe,SForge 只传显式选择。直接用全局聚合结果可少一次遍历,但一个无关 Todo 就能占据唯一的 latest 或 previous_best。

具体改动 / Changes

评审 head be1e72c,目标和 merge base 96a3035。首次观察的 main 为 b9faf8d,实际目标仍是 r35。GitHub 与本地全 diff 路径一致,共 238 文件 +20,442/−2,280。按 main 同步、C1 合入和后续组件分层核对。C1 的手工合并主要触及 quota_capture、progressive_context 与 codex_context。组件原有结论用于定位不变的 producer,并复验实际集成消费者。 C1 组件 · 量测比较 · 原捕获 · probe · 接入

关键代码讲解

  • capture_before_replan 用 prepared-capture-reference 取摘要并核验一次原 guard,native reader 继续从同一份 decision.json 展开。显式关闭走原 view。 checkpoint
  • projectExploreTurnContext 先对全局 findings 按 node 和量测键聚合。measuredSearchComparison 再检查所得 pair 是否属于当前 Todo,已经被覆盖的原 pair 无法取回。 读取路径 · 比较函数
  • run_probe 保存候选绑定与主执行结果。Linux 的顺序 PSS 估计是可选附件,侧车失败不应改变原结果。BenchmarkCodex.install 按选择用隔离解释器运行暂存入口,检查完成才进入 worker。 执行观察 · 安装

以 docs/architecture/rfcs/controller-executor-unified-contract-v0.md 的 96a3035 版本为准。Lossless context delivery 已覆盖原 guard、scope 与重放路径,native fork 的共同失败单独保留。One complete contract 的剩余产品消费者和默认退休继续由 C1 负责。Shared production paths 的同 Todo 读回目前有反例。Outcome and cost benefit 仍需匹配研究。 基线规范

对主干的风险 / Risks

SQLite 的真实 quota、refresh-state、spend 和 Explore 读回,在同一 Todo 产出 8 分与 7 分时先返回 8 分候选。 另一个已认领的 Todo 在同一 node 和量测范围写入 9 分后,原 Todo 仍保留原 finding 与 next_action,但 search_comparison 和紧凑 comparison_cue 消失。 base 没有新比较字段,公共 measurement_feedback 与原作者建议仍保留。建议按同一 node 与原 binding 过滤后调用原聚合器,重新跑两个 Todo 的普通及详情读回。当前单 Todo 和 off 的普通回归可通过。 作用域契约

真实原入口在两个版本均有健康对照。最终回执替换前失败和替换后失响应两个故障都丢失 managed_run 与 solver 摘要,原试验 replay 返回 2,原槽位未释放。 这是原样继承的 r35 风险,和新增的比较问题分别记录。以本机暂存的默认 SDK 和实际 controller 原入口复现,执行与评分使用受控数据。 前序问题 · 现有 fallback

实装 Codex CLI 通过本地确定性 provider 执行,无外部模型调用。head 有 11 项通过、1 项失败、2 项 supervisor 用例跳过。base 有 10 项通过,另有显式 off 测试的旧默认行为失败。共同失败为 native fork 第三个请求未见 heartbeat_receipt。 quick premerge 的 7 个直接和 4 个选中检查均无机械失败,含双语 RFC 与公开扫描。benchmark-sensitive 人工门仍在。 C1 相关测试 413 passed in 547.65s (0:09:07)。 固定 SDK、隔离代理变量后的 SForge/Harbor 套件。head 447 passed in 246.51s (0:04:06),base 441 passed in 246.50s (0:04:06)。 聚焦 Python 初跑缺 SDK,相关用例跳过。补齐固定 SDK 后先遇到继承的 SOCKS 环境,再在两个版本移除这些变量用同一套件复测。上下文传输字节不推导模型费用或得分。

语义与 CI 对齐

新增的比较提示声明与代码投影应同为 Goal/Agent/Todo 作用域。现有 state 与测量类型可复用,先切同域输入再派生 cue。完成反例后在实际 SQLite 的两个工作上重跑。远端 CI 独立于本地评审。

我的整体评价 / Verdict

当前提交请求修改。同一工作的匹配证据应当跨其他 Todo 的追加记录保持可用。先修这处新比较,再检查原试验终态恢复。C1 读回和可选 probe 有可保留的组件证据,本轮没有运行完整镜像、实际安装或自然模型,继续把长程效果和总成本留给匹配验证。

English verdict: REQUEST_CHANGES for be1e72c. A sibling Todo can erase an otherwise valid scoped comparison. The managed terminal receipt recovery failure also reproduces at both revisions.

row.node_id === source.node_id && profiles.some(p => matchesProfile(row, p)));
const scopedFindings = new Set(findings.filter(row => row.node_id === source.node_id && belongsToWork(row, params))
.map(row => row.finding_id));
const searchComparison = measuredSearchComparison(feedback, decision, scopedFindings);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Aggregate comparisons within the selected Todo

另一个已认领的 Todo 在同一 node 和量测范围写入 9 分后,原 Todo 仍保留原 finding 与 next_action,但 search_comparison 和紧凑 comparison_cue 消失。 按 Goal、Agent、Todo 和 node 过滤原 findings 后,复用既有量测聚合器计算比较。全局量测列表继续沿原路径。

@loopx-agent loopx-agent closed this Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants