Repository navigation
eval: 召回判断 high/low 对照与预算建议(RJ-B1) - #123
Merged
Merged
Conversation
leafliber
added a commit
that referenced
this pull request
Oct 17, 2026
更新 2026-10-17「上线测试第一周:日志分析与修复」一节: - 加入 #123 的召回判断 high/low 评测结论与推荐配置。 - 记录宿主预算超时计入用途暂停的已知问题及上线观察安排。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
上线首周召回判断在插件 5 秒预算下有 34% 超时。本 PR 只增加 RJ-B1 汇总报告,比较七份公开语料的 high/low:两档各 191 次判断全部成功,low 的 P95 从 3.657 秒降至 2.214 秒,各语料 Recall 与 relevant 精确率均不下降,误返均为 0;合计 nDCG 小降 0.0013,符合现有守则。
建议仅将 recall_judge 设为 low,插件使用 judge_budget_seconds=7、prepare_timeout=12。报告说明 5/7 秒超时数是同轮耗时估算、10 秒为实测,并用四个假网络场景证明宿主短预算到期会触发三次错误暂停,提出区分超时来源的后续实现建议。本次不修改产品代码或线上设置,无作废运行。
完整材料:/Users/cassia/Local/Code/iris-eval-artifacts/RJ-20261017/。旧 M2 材料和缓存缺失,本次重新生成 471 个 embedding。191 对请求除档位外完全一致。
验证:Python 3.13 与隔离 Python 3.12 全套测试各 3713 passed;自己的 .pytest-tmp 已清理。仅提交 evals/reports/recall-judge-effort-20261017.md。