[hidb4ai]

Which issue or RFC does this PR close?

Adds the input-encoding evaluation workflow for RFC #2086. Replaces the corresponding scope of #2264.

Depends on #2311, which includes the input-encoding implementation merged from #2314, and targets its branch so this diff contains only the evaluation workflow. After #2311 merges into master, retarget this PR to master and refresh the branch before marking it ready; do not merge into the feature base branch.

Rationale for this change

Smaller reversible inputs alone do not demonstrate useful Experience output or lower total model cost. This harness compares off/shadow/active using identical public evidence, preserves failed and unknown outcomes, and separates offline representation measurements from live quality and usage evaluation. A separate fixed synthetic probe checks what a model infers directly from the selected representation; Python reconstruction is not used as evidence of model comprehension.

What changes are included in this PR?

• Add a resumable evaluation CLI using the pinned 731-record SWE-bench Pro public evidence corpus and a fixed live-evaluation slice.
• Exercise Source ingestion, Experience generation, approval, lineage readback, and restart verification; freeze experiment identity and independent reference questions for grading. Reject Anthropic profile discovery that cannot be bound to the experiment identity.
• Report quality, failures, input/output usage, unknown billing, latency, and exploratory paired intervals without treating historical patches as newly solved tasks. Failed producer cases remain in the planned denominator while successful cases continue through automatic judging.
• Add a direct-model semantic probe with seven fixed synthetic case groups and 29 questions covering path roles, literal placeholder markers, Source/Scope attribution, repeated events, JSON-string types, and exact whitespace. Exercise both codecs across off/shadow/active, with targeted encoding coverage and fixed expected answers kept out of model input.
• Preserve probe identity and saved outputs on resume, regrade actual answers, report all errors and unknown usage, and keep existing attempts from being automatically replayed. Add CI path selection for evaluator sources and fixtures.
• Add harness/identity/judging/resume regressions and documentation. Keep the optional database/provider connectivity probe explicitly separate from encoding acceptance; database baseline inspection does not provision an absent OceanBase database.

Are there any user-facing changes?

This adds an opt-in developer evaluation workflow and makes no production API, database schema, or default inference behavior change. Live Experience runs require explicit provider/database configuration and can incur model usage. The standalone semantic probe requires explicit model configuration for live runs, uses no tools or database, and makes at most one request per attempt. Its dry-run only validates the case plan and encoding coverage. Passing the fixed cases does not establish semantic equivalence or acceptance for all four Runtime operations.

Identity-backed live commands (run, judge without --validate-only, and semantic-probe without --dry-run) require POSIX private-file support. Windows live execution is explicitly rejected before private configuration reads or run-file writes; offline, report, judge --validate-only, and semantic-probe --dry-run remain available. Owner-only key permissions and link protections are preserved.

How was this change tested?

• uv run --locked pytest -q tests/evaluation/input_encoding: 115 passed on Linux. An identity-capability simulation on the same host passed 59 tests and skipped 56 identity-backed live cases; this is not native Windows acceptance.
• The pinned 39-chunk failed case no longer aborts grading of a successful case in the same cohort (controlled model adapter). Controlled database-driver tests verify missing databases are not created; no external OceanBase service was used.
• Semantic dry-run: seven case groups, 29 questions per schema/mode, 42 planned attempts; all required encoded fields covered. No model calls or private configuration access.
• uv run --locked prek run -a: passed, including lint, formatting, and type checks.
• Both CLI help entry points passed.
• Full pinned-corpus offline run: 731 records accounted for, 730 measured and one rejected at the production projection boundary. All measured modes reconstructed exactly; active selected 123 candidates, with 606 insufficient-savings fallbacks and one input-limit fallback.
• No live model quality, billing, external database, or upstream repository test result is claimed by these offline checks.

AI usage statement

OpenAI Codex assisted with evaluation implementation, regression tests, scope separation, documentation, and validation.

[hidb4ai]

Reviewed 5e3dcfe. Three correctness issues remain in configuration identity, the read-only database probe, and failure isolation during automatic judging. These should be fixed before merge; details are inline.

[hireview001]

• CI failed: tests (3.12, unit) (failure).
• CI failed: tests (3.11, unit) (failure).
• CI failed: tests (3.14, unit) (failure).
• CI failed: tests (3.13, unit) (failure).
• CI failed: Main (failure). Failed steps: tests (3.13, unit) / Run unit tests; tests (3.14, unit) / Run unit tests; tests (3.11, unit) / Run unit tests.

[knqiufan]

Reviewed 7c6939d. The missing-external-database issue already has a review thread and is not duplicated. One additional configuration-input regression is inline.

[AlexStocks]

[P2] p50 uses two different estimators in adjacent sections

elapsed_seconds.p50 and generation_seconds.p50 use statistics.median, while queue_wait_seconds.p50 (line 126) and queue_inclusive_seconds.p50 (line 131) use percentile(..., 0.50) → nearest-rank. The two definitions disagree whenever the sample count is even: for [1, 2, 3, 4], median returns 2.5 and percentile(..., 0.50) returns ordered[ceil(4 * 0.5) - 1] = 1. With the default --limit 24 × --repeats 2 × 3 modes = 144 rows, and per-mode subsets of 48, every section in one report is computed on an even count, so the readable "p50" of elapsed time is not the same statistic as the "p50" of queue wait two lines below.

Since the report is the artifact these runs are read from, either estimator is fine but the report should use one. percentile(elapsed, 0.50) and percentile(generation, 0.50) would make all four sections consistent and keep the existing nearest-rank semantics that p95/p99 already use.

[AlexStocks]

在 18724d30 上复核完毕。提醒一下 head 在我审查期间从 7c6939d9 漂到 18724d30（d7689f49 / a38358b0 / 027abce1 / 15c56127 / 18724d30），之前挂在线程里的「Fixed in …」在新 head 上确实成立，逐条验证如下。

已确认落地的线程

• identity.py:78 新增 supports_private_identity() / require_private_identity_support()，并接进 run()、prepare_configuration_identity() 和 __main__（parser.error 输出可操作提示）；tests/evaluation/input_encoding/ 全部加了 skipif(not supports_private_identity())。
• runner.py:_configuration 改为 BuiltinConfig(database=json.loads(...)).database，残缺 mapping 继承 SQLite 默认值，不再 KeyError: 'kind'。
• _open_runtime 用 OceanBaseProfile.open_existing 预检外部库；judging.py 只对 status == "succeeded" 的 case 出题并透传 streaming；source_hash() 纳入 catalog.py / adapter.py。

本地验证（新 head）

• WSL（POSIX）：tests/evaluation/input_encoding → 115 passed
• Windows：同目录 → 59 passed, 56 skipped, 0 failed（旧 head 是 46 failed，全部由 os.O_NOFOLLOW / os.getuid 缺失引起，现在被显式拒绝并跳过）

本轮新增一条：report.py:115 / :121 的 p50 用 statistics.median，而 :126 / :131 用 percentile(..., 0.50)，偶数样本下同一份报告里两个同名统计量不一致，见新开的行内评论。

其余部分（dataset.py 的 12 KiB 分片与 write_json 原子替换、offline() 的 roundtrip 断言、judging.py 的 reference-question 校验与 excerpt 逐字比对、semantics.py 的 dry-run 覆盖率闸门）未发现可归因缺陷。