Skip to content

Measure source reads and graph outcomes along namesake adjudication paths - #1377

Merged
WaylandYang merged 5 commits into
deeplethe:devfrom
virtue039:codex/adjudication-evidence-measurement
Oct 29, 2026
Merged

WaylandYang merged 5 commits into
deeplethe:devfrom
virtue039:codex/adjudication-evidence-measurement

Conversation

@virtue039

@virtue039 virtue039 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

An exact-name same can reach an automatic merge without reading either source. This test-only measurement for #1193 records whether investigation runs, what each side actually receives, and the resulting graph/review state.

Changes requested in review

Addresses review 5658119082:

  • Give every row a stable name such as Batch/ordinary/forward/distinct. Explicitly list today's 20 wrong merges and four unmerged true pairs in KNOWN_GOLD_ERRORS. Each listed row must still disagree with source gold; every other row must match it. Unexpected passes, unlisted errors, duplicate/missing rows and unused list names fail the test. There is no total-error budget.
  • Keep the original eight inputs/gold, all 34 existing checks and the original empty KNOWN_FAILURES list intact. The new named list applies only to the new 48-row measurement. A future policy fix removes its corresponding names with the fix.
  • Convert all newly added comments to Chinese.

The 48 cases cover ordinary adjudication/governance, both creation orders, source-defined same/distinct pairs, batch same(0.8) and same(0.9), a seeded ordinary verdict cache, facts/quotes on both sides and a final different(0.5). Governance does not read the ordinary cache. The assertions also check A/B ownership in real HTTP inputs/tool results, verbatim source delivery, graph/review agreement and no repeated work on a terminal pair.

Scripted conclusions deliberately disagree with some gold labels. Green CI means exactly the named baseline errors remain and controls match gold; it does not mean these identity errors are fixed or estimate provider accuracy. No production gate, prompt, API, schema, model request or lifecycle state is added by this PR.

Calibration and validation

A temporary mock-reply calibration made Batch/ordinary/forward/distinct correct and made the true-pair control Batch/ordinary/forward/same incorrect. All 48 ran and the gold-error total stayed 24, yet the test failed on both the unexpected pass and unlisted error. The temporary reply change was restored before final verification; source inputs, gold labels, lists and production code were unchanged by the calibration.

At signed commit f652023197b051fe3a27406da7ae0fce26eebd35, based on dev 1142792979b61272d92cf322cbfd2f86615f6cb3 (2026-10-27), on macOS arm64:

  • Workspace: 1,797 passed / 0 failed / 6 existing ignored, with DB/PDF require guards enabled and an isolated migrated PostgreSQL database.
  • All 34 existing checks and all 48 new rows complete; the underlying observations match the pre-review measurement, with stable row IDs added.
  • Formatting, locked workspace/all-target clippy (-D warnings), workspace build and frozen frontend install/build pass.
  • The initial local run stopped because existing PostgreSQL client tools were not on PATH for the new upstream CLI restore test. After adding the installed 16.15 clients, the real restore test and complete guarded workspace suite pass. The initial log is retained.
  • All normal measurement requests use the local scripted service; no paid provider calls were made.

CI 40486272486 passes all six checks at signed commit f652023197b051fe3a27406da7ae0fce26eebd35. The PR remains ready for the maintainer's re-review; the previous changes-requested review has not been dismissed.

Landing order

#1425 was still open/unmerged when this revision was validated and submitted. Its implementation is not included here. As requested in review, whichever PR lands second must rerun this measurement on the integrated tree.

#1251 is already merged. The production evidence standard remains open under #1193; this PR establishes the required named baseline guard.

Refs #1193, #1136, #1202, #1220.

Integration preflight (2026-10-28)

A local merge of this head f652023197b051fe3a27406da7ae0fce26eebd35 with #1425 head 2916e332150bc5baa1435c0a0fd14721bf85f899 merges without conflicts. The tested local tree is 80facb8ed2df69589d8dacf10b16a2e746eb7fb5 (local preflight commit a0dc25bd51fbdfb9a455ff10067325124bc91b2e, not pushed).

  • Full Rust workspace: 1,800 passed / 0 failed / 6 existing ignored, with isolated migrated PostgreSQL and DB/PDF require guards.
  • All 34 original checks and all 48 decision-path rows pass their baseline constraints. Every observation matches this PR's original measurement, including exactly the same 24 named gold errors.
  • All three Merges involving names declared as identities wait for human review #1425 identity-source safeguard tests pass on the combined tree.
  • Formatting, locked workspace/all-target clippy (-D warnings), workspace build, frozen frontend install, 338 web tests across 49 files, and frontend build pass.
  • No new provider requests were made. This verifies engineering compatibility, not a model quality improvement.

This pre-merge check does not replace the landing requirement: whichever PR lands second must rerun the measurement on the actual integrated upstream commit. Neither PR was merged by this preflight; the previous changes-requested review remains for the maintainer to re-review.

…nges the graph

Signed-off-by: Virtue039 <virtue039@qq.com>
… behavior

Signed-off-by: Virtue039 <virtue039@qq.com>

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — this is a clean, genuinely test-only measurement, and CI cost is negligible. One change before merging: please pin the outcome of each of the 48 rows the way namesake_scripted_cases.rs does. Name today's 20 wrong merges and 4 unmerged true pairs in a known-gold-error list (e.g. Batch/ordinary/forward/distinct), make listed rows required to show the error and all others required to match gold. That way a policy fix must update the list and a regression can't pass silently. Please also switch the new comments to Chinese per repo convention. #1425 doesn't overlap textually, but whichever lands second should rerun this.

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — the 24 gold errors are now pinned by name and the comments are in Chinese. #1425 has landed, so CI on the updated branch is the rerun on the integrated tree. Merging.

@WaylandYang
WaylandYang merged commit 1d94b53 into deeplethe:dev Oct 29, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants