Skip to content

A second, independent check that an explanation hasn't drifted #410

Open

Description

@patelankeet2

As a maintainer, I want a second, independent signal on whether an explanation has drifted from its source finding, so that a hallucination is more likely to be caught before it reaches a user.

Acceptance criteria

  • A second check (for example a semantic-similarity comparison between the generated explanation and the finding's own evidence.detail) runs alongside the existing held-out scoring, not instead of it.
  • A low-similarity result is surfaced as a measurable signal in the quality-scoring report, not silently dropped.
  • The check adds no new failure mode to the live /api/findings response path - it is a CI and offline measurement tool first, not a runtime gate.

Shape

AI layer, tooling. Extends #385/#407's existing scoring infrastructure rather than replacing it.

Why this is worth doing

The current scoring is text-matching based. An independent signal using a different method (semantic similarity rather than substring matching) would catch a different class of drift than either existing tool catches alone - the two methods have different blind spots.


Epic: Candidate idea (AI layer) - not yet agreed as a specialist pick
Size: not yet estimated
Assigned: @patelankeet2

Proposed 27 October 2026 as a candidate feature idea. Not yet discussed or agreed by the team; open for refinement or rejection at the next planning session, the same status every other idea in this batch carries.

Activity

  1. self-assigned this
    on Oct 26, 2026
  2. added
    enhancementNew feature or request
    layer: aiLayer 2 — local LLM explanation and Q&A
    phase: 3Market readiness -- what stands between us and a product someone could buy.
    on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestlayer: aiLayer 2 — local LLM explanation and Q&Aphase: 3Market readiness -- what stands between us and a product someone could buy.

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions