Skip to content

eval: 召回判断 high/low 对照与预算建议(RJ-B1) - #123

Merged
leafliber merged 1 commit into
mainfrom
eval/recall-judge-effort
Oct 17, 2026
Merged

leafliber merged 1 commit into
mainfrom
eval/recall-judge-effort

Conversation

@leafliber

Copy link
Copy Markdown
Member

上线首周召回判断在插件 5 秒预算下有 34% 超时。本 PR 只增加 RJ-B1 汇总报告,比较七份公开语料的 high/low:两档各 191 次判断全部成功,low 的 P95 从 3.657 秒降至 2.214 秒,各语料 Recall 与 relevant 精确率均不下降,误返均为 0;合计 nDCG 小降 0.0013,符合现有守则。

建议仅将 recall_judge 设为 low,插件使用 judge_budget_seconds=7、prepare_timeout=12。报告说明 5/7 秒超时数是同轮耗时估算、10 秒为实测,并用四个假网络场景证明宿主短预算到期会触发三次错误暂停,提出区分超时来源的后续实现建议。本次不修改产品代码或线上设置,无作废运行。

完整材料:/Users/cassia/Local/Code/iris-eval-artifacts/RJ-20261017/。旧 M2 材料和缓存缺失,本次重新生成 471 个 embedding。191 对请求除档位外完全一致。

验证:Python 3.13 与隔离 Python 3.12 全套测试各 3713 passed;自己的 .pytest-tmp 已清理。仅提交 evals/reports/recall-judge-effort-20261017.md。

@leafliber
leafliber merged commit 203ecc9 into main Oct 17, 2026
1 check passed
@leafliber
leafliber deleted the eval/recall-judge-effort branch October 17, 2026 11:54
leafliber added a commit that referenced this pull request Oct 17, 2026
更新 2026-10-17「上线测试第一周:日志分析与修复」一节:
- 加入 #123 的召回判断 high/low 评测结论与推荐配置。
- 记录宿主预算超时计入用途暂停的已知问题及上线观察安排。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant