Skip to content

Audit V5/V6 accuracy failures with paired ranking evidence and strict drift checks - #14

Merged
yashaskn8 merged 10 commits into
mainfrom
feature/servy-accuracy-evidence-gates
Oct 19, 2026
Merged

yashaskn8 merged 10 commits into
mainfrom
feature/servy-accuracy-evidence-gates

Conversation

@yashaskn8

Copy link
Copy Markdown
Owner

Source-verified causes addressed

  • V6 previously kept only aggregate retrieval metrics; equal totals obscured query-level regressions.
  • Hit@1 under head-five reranking is bounded by the fraction of queries whose gold evidence is in the first five frozen candidates.
  • Development-only comparisons cannot prove production accuracy or independent holdout performance.

Changes

  • Add pure CPU-only paired diagnostics with candidate-recall/top-one oracle ceilings, query-level movement, and strict frozen/gold/rank checks.
  • Save per-query reports and SHA-256 fingerprints for V5/V6 comparisons to existing approved private Colab output folder.
  • Run counterfactual pool-size 5/8/10 analysis using existing scores with no changes to serving.
  • Add regression tests and CI gate, plus a no-retraining historical replay runbook.

Boundaries

No model training, no private weights/data uploaded, no automatic model promotion, no change to production retrieval/authorization. CI is synthetic. The previously completed experiment requires a fresh read-only replay to obtain the new per-query artifacts.

Required verification

CI backend suite (MySQL) and frontend build/lint; private V5/V6 reproducible inference, candidate-set hashes and eventual independent holdout are not verified in CI.

@yashaskn8
yashaskn8 merged commit 1e747e0 into main Oct 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant