Skip to content

Keep distinct source quotes available to identity adjudication - #1457

Draft
virtue039 wants to merge 2 commits into
deeplethe:devfrom
virtue039:codex/preserve-adjudication-quotes
Draft

virtue039 wants to merge 2 commits into
deeplethe:devfrom
virtue039:codex/preserve-adjudication-quotes

Conversation

@virtue039

@virtue039 virtue039 commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

The adjudicator's quotes tool silently discarded every other quotation from the same chunk. In a recorded namesake failure, the graph contained both the finance director's management-award sentence and the engineer's technical-award sentence, but repeated source lookups returned only the first sentence.

Deduplicate by chunk and rendered quotation instead. Return the first passage from each chunk before additional passages, retain ordinary-fact priority over name evidence, and keep the existing caller limit (six passages) and output clipping. Duplicate facts, invalidated facts and bare-name evidence do not consume additional slots. No model prompt, adjudication trigger, confidence gate, schema or extraction protocol changes.

Validation on macOS arm64, current head 82c60d530f920183c336aa28129b56a1f4b6fbe6, merged with dev 1d94b532860ba7e60888f75449367da9388b1ab0 after #1425 and #1377 landed:

  • The amended name-evidence regression and new distinct-quote regression fail on the unchanged query and pass with this fix. Both insertion orders, duplicate facts, subject/object lookup, invalidation, KB isolation, deterministic ordering and the six-source bound are exercised.
  • Revalidated after synchronization with required database/PDF checks: 1,808 Rust tests passed, zero failed, 6 existing ignored. Migration, format, workspace clippy/build, frontend 338 tests and build passed.
  • All six CI checks passed on current head 82c60d5 (run 41011095162).
  • On the actual integrated head, all 48 adjudication observations remain unchanged, including the 24 named gold errors. The earlier isolated combination with Recover grounded statement subjects omitted from the entity list #1456 additionally retained the original 34 checks and all 80 recovery observations. That combination is not a model-quality result.
  • Frozen historical downstream control: the original pre-investigation graph, initial batch pair and six unchanged recorded replies run through the real investigation tools and apply_verdict. Initial request messages match the recording on both builds. Before: one award quote. After: both award quotes reach the actual model request. Both runs make six scripted calls.

Fresh investigation pilot (not a corpus benchmark): fixed qwen3.7-flash-2026-07-15, thinking disabled, temperature 0; four restored-state investigations in baseline/fix/fix/baseline order with identical initial requests. Both baseline runs merged at 0.95. The fix runs yielded one merge at 0.95 and one different(0.5) left pending human review. All four runs skipped the quotes tool, so the observed variation cannot be credited to this query fix. Seven successful calls, 18,350 tokens; connectivity preflight 17 tokens separate. All results retained, no selective reruns. This pilot froze the earlier heads 1142792 / a6b758d; it is not a measurement of the newly integrated head.

This does not establish an identity-quality improvement. Replaying the original erroneous same(1.0) response still merges in both versions; the control is intentionally not changed to a successful answer. The upstream name-only extraction has already put two people's awards on one entity, which this source-retrieval fix cannot split. No 16-KB comparison has been run for this change. Additional passages may increase input tokens within the existing output bound. Keep draft until the remaining evaluation/review questions are resolved.

Related: #1193, #1136. Neither issue is closed by this PR. #1456 remains a separate failed-acceptance draft (wrong merges 15→23); this patch does not rehabilitate its measurements.

  • All commits signed off
  • Tests exercise the production SQL and run in normal workspace CI
  • Reproduce the source omission on the actual failure recording
  • Confirm real-model behavior before claiming fewer false merges

Signed-off-by: Virtue039 <virtue039@qq.com>
…ation-quotes

Signed-off-by: Virtue039 <virtue039@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant