diff --git a/evals/entry_profile_eval/EXP2-report.md b/evals/entry_profile_eval/EXP2-report.md new file mode 100644 index 00000000..d601a444 --- /dev/null +++ b/evals/entry_profile_eval/EXP2-report.md @@ -0,0 +1,150 @@ +EF-29 拆分说明：以下为历史 EXP2 的汇总。用户决定入口画像维持 main；本 PR 仅保留重复运行、离线汇总与配对工具，不包含提名并集设置或产品逻辑。完整运行与判分材料继续保留在文末仓库外目录。 + +EF-28b / EXP2：公开 entry_profile_v2 提名并集发现实验 + +本次提名并集没有提高最终可用记忆的平均覆盖：A 为 40.33/74，B 为 39.00/74，B−A 为 -1.33 个要点（-1.80 个百分点）。12 例中 3 例改善、7 例下降、2 例持平。实际 GLM 调用量为 A 的 1.828 倍，运行墙钟时间为 2.360 倍。本次结果不支持默认开启提名并集。 + +代码和方法 + +两组均冻结在 PR #128 的 94a7778ab31236a1178d675bf18f3c2e79277765；A 关闭提名并集，B 开启。语料冻结于 main 93d1be9，JSON SHA-256 为 7a434db4c92923ff6e6ec09f056d3f13f7ba3c45eaa1b4b7d53cc3e4fbc26f77（12 例、966 条消息、74 个 must_cover、136 个 forbidden 标签）。使用 memory_runner.run_repeated，即 CLI repeat --runs 3 的同一实现；外部控制器依次启动完整 A 组和完整 B 组，replay_mode=daily_plus_final_flush，GLM high，每次独立数据库和目录。 + +每次有效运行由一个全新、未参与实现的 gpt-6.1-sol max 子代理单判，共六名，fork_turns=none。判分材料使用中性随机编号，移除运行版本／组别元数据；72 份 input 与原始导出完全相同。各判分员只访问自己的材料和结果目录。结果原样接回原始指纹后由冻结的 score_draft、summarize、compare 验证和汇总。全部六次有效运行均计入。scoring_v3、计分合并、gate_eligible 与作废规则均未修改；重复／配对汇总按原实现保持 descriptive_only=true、gate_eligible=false，本报告不作门槛通过声明。 + +逐次结果。违规记忆按不同记忆计数，包含“无依据”，不重复计算同一记忆的宿主投影；禁止列为 forbidden 标签命中数。覆盖列沿用 must_cover，分母均为 74。 + +| 运行 | 模型记忆有依据 | 违规记忆 | 禁止 | 覆盖 | 实际调用 | 墙钟分钟 | +| --- | --- | --- | --- | --- | --- | --- | +| A1 | 43/44（97.73%） | 2 | 1 | 38/74 | 291 | 78.39 | +| A2 | 51/57（89.47%） | 6 | 0 | 48/74 | 284 | 74.16 | +| A3 | 39/44（88.64%） | 5 | 3 | 35/74 | 260 | 69.60 | +| B1 | 79/81（97.53%） | 2 | 0 | 50/74 | 546 | 176.51 | +| B2 | 58/58（100.00%） | 0 | 0 | 34/74 | 449 | 147.89 | +| B3 | 53/76（69.74%） | 24 | 2 | 33/74 | 531 | 199.86 | + +A／B 三次均值如下。有依据比例使用每次比例的算术平均；汇总记忆口径另列，避免混用分母。 + +| 指标 | A 三次均值 | B 三次均值 | +| --- | --- | --- | +| 最终覆盖 | 40.33 | 39.00 | +| 模型记忆数 | 48.33 | 71.67 | +| 有依据模型记忆数 | 44.33 | 63.33 | +| 有依据比例（逐次比例均值） | 91.95% | 89.09% | +| 不同违规记忆数 | 4.33 | 8.67 | +| 禁止标签命中数 | 1.33 | 0.67 | +| 实际 GLM 调用数 | 278.33 | 508.67 | +| 墙钟分钟 | 74.05 | 174.75 | + +三次合计的模型记忆有依据为 A 133/145（91.72%）、B 190/215（88.37%）。 + +逐案例配对。向量按各组第 1／2／3 次运行排序，均值差为 B−A；按同案例三次平均比较，非按最佳运行比较。 + +| 案例 | 要点数 | A 三次覆盖 | A 均值 | B 三次覆盖 | B 均值 | 均值差 | +| --- | --- | --- | --- | --- | --- | --- | +| EP101 | 7 | 6／5／4 | 5.00 | 6／4／4 | 4.67 | -0.33 | +| EP102 | 6 | 3／3／2 | 2.67 | 5／3／5 | 4.33 | +1.67 | +| EP103 | 6 | 3／6／5 | 4.67 | 5／5／4 | 4.67 | +0.00 | +| EP104 | 6 | 2／5／4 | 3.67 | 4／1／4 | 3.00 | -0.67 | +| EP105 | 6 | 2／2／1 | 1.67 | 2／3／1 | 2.00 | +0.33 | +| EP106 | 6 | 1／5／3 | 3.00 | 4／2／2 | 2.67 | -0.33 | +| EP107 | 6 | 2／3／2 | 2.33 | 2／2／1 | 1.67 | -0.67 | +| EP108 | 6 | 3／3／2 | 2.67 | 4／2／3 | 3.00 | +0.33 | +| EP109 | 6 | 5／3／4 | 4.00 | 4／5／3 | 4.00 | +0.00 | +| EP110 | 7 | 6／4／3 | 4.33 | 5／3／2 | 3.33 | -1.00 | +| EP111 | 6 | 2／4／3 | 3.00 | 5／1／2 | 2.67 | -0.33 | +| EP112 | 6 | 3／5／2 | 3.33 | 4／3／2 | 3.00 | -0.33 | + +A 三次覆盖为 38／48／35，均值 40.33，范围 35—48，极差 13，样本标准差 6.81 个要点。B 三次为 50／34／33，范围 33—50、极差 17。每次由不同单判员判分，观察到的波动混合了生成随机性和单判差异；本轮不能分离两者。 + +违规与禁止的汇总保留两种既有计数，避免投影重复改变印象。两个相加口径都可能重复命中同一问题，不能解释为不同问题总数。 + +| 三次合计口径 | A | B | +| --- | --- | --- | +| 不同违规记忆（含无依据） | 13 | 26 | +| forbidden 标签命中 | 4 | 2 | +| 不同违规记忆＋forbidden | 17 | 28 | +| 原始违规标签命中（含无依据及投影重复） | 45 | 46 | +| 其中投影违规标签命中 | 22 | 16 | +| 原始违规标签＋forbidden | 49 | 48 | +| 排除无依据后的禁止类违规标签（仍含投影重复） | 22 | 10 | + +阶段覆盖与发现缺口。各阶段均按冻结标签的完整覆盖判定；生成阶段 false 不等于完全没有相关候选，也可能是候选类别／正文／证据尚不足。因此以下“发现缺口”是生成阶段尚未完整覆盖的标签数，不新增评分口径。 + +| 运行 | 生成 | 确定性后／送检 | 检查通过 | 宿主投影 | 生成→送检丢失 | 送检→检查丢失 | 检查→投影丢失 | +| --- | --- | --- | --- | --- | --- | --- | --- | +| A1 | 49 | 45 | 38 | 37 | 5 | 7 | 1 | +| A2 | 56 | 54 | 48 | 44 | 4 | 6 | 4 | +| A3 | 50 | 40 | 35 | 34 | 12 | 5 | 1 | +| B1 | 65 | 58 | 50 | 36 | 7 | 8 | 14 | +| B2 | 53 | 39 | 34 | 29 | 14 | 5 | 5 | +| B3 | 58 | 43 | 33 | 27 | 17 | 10 | 6 | + +| 阶段覆盖均值 | A | B | B−A | +| --- | --- | --- | --- | +| generated | 51.67 | 58.67 | +7.00 | +| deterministic | 46.33 | 46.67 | +0.33 | +| checked | 40.33 | 39.00 | -1.33 | +| projected | 38.33 | 30.67 | -7.67 | + +阶段丢失按逐标签 True→False 计，生成→送检还存在不同材料组合带来的 False→True：A 三次新增为 1／2／2，B 为 0／0／2。因此不能直接相减阶段总数来替代丢失标签数。 + +每次平均的生成阶段缺口由 A 22.33 减为 B 15.33，少 7.00 个；三次生成阶段均未完整覆盖的标签由 12 个减为 6 个：原 A 缺口中有 7 个在 B 至少出现一次完整覆盖，同时有 1 个成为 B 新缺口。生成阶段增量未转化成最终可用记忆的覆盖增量；投影覆盖均值还由 38.33 降到 30.67。 + +以下编号冒号后的数字为该案例冻结 must_cover 数组的 1 起始序号，未修改语料标签。 + +| 发现缺口变化 | 标签编号 | 冻结公开标签 | +| --- | --- | --- | +| A 三次均缺、B 至少一次完整覆盖 | EP104:05 | 群内说法“空栏有名”：本项须覆盖说法及缺测格留空、写清中断原因两个释义要点；只写词面不算覆盖。释义依据须含“空栏有名就是缺测格留空，同时写清中断原因”原话。 | +| A 三次均缺、B 至少一次完整覆盖 | EP105:03 | 常聊话题：短波接收中不同频段、收听时段与信号衰落快慢的对照。依据包括“把频段和收听时段放一块”“声音一阵浮上来一阵沉下去”“频段、时段写在旁边，衰落快慢才对得上”；须覆盖时段与频段的对应及强弱起伏，不要求写某日某频段的结果。 | +| A 三次均缺、B 至少一次完整覆盖 | EP106:01 | 常聊话题：熔融打印首层的附着、线条相连情况，以及首层挤压对底部边缘尺寸的影响。 | +| A 三次均缺、B 至少一次完整覆盖 | EP107:04 | 群内说法“收一道岸”：本项须覆盖说法及把遮盖边沿再压贴一遍的含义；只写词面不算覆盖。释义依据须含“收一道岸，就是把遮盖边沿再压贴一遍”原话。 | +| A 三次均缺、B 至少一次完整覆盖 | EP108:05 | 直播间说法“托住根”：本项须覆盖说法及让支撑落在完整围岩处、避开有裂隙晶尖两个释义要点；只写词面不算覆盖。释义依据须含“托住根就是让支撑落在完整的围岩处，避开有裂隙的晶尖”原话。 | +| A 三次均缺、B 至少一次完整覆盖 | EP111:04 | 群内说法“掐掉伴奏”：本项须覆盖说法及讨论结论时先拿开配乐情绪、再看话本身的材料支持两个释义要点；只写词面不算覆盖。释义依据须含“先把配乐带来的情绪拿开”和“再看话本身靠哪段材料站住”两处原话。 | +| A 三次均缺、B 至少一次完整覆盖 | EP112:03 | 常聊话题：用正面和侧面镜头互补检查躯干姿态，正面看左右偏移、侧面看前倾或前后关系。依据包括“看躯干左右偏不偏”“正面查左右偏移，侧面查前后关系”和“正面看左右偏，侧面看前倾”。两个姿态维度均须覆盖，不泛指一切镜头观察。 | +| A 至少一次完整覆盖、B 三次均缺 | EP107:05 | 群内说法“留口气”：本项须覆盖说法及掉漆痕迹之间留干净间隔的含义；只写词面不算覆盖。释义依据须含“留口气指掉漆痕迹之间留干净间隔”原话。 | + +B 三次生成阶段仍均未完整覆盖：EP105:01、EP105:05、EP107:01、EP107:05、EP108:03、EP110:05。全部 74 个标签在两组各三次、四个阶段及最终可用记忆中的布尔结果见 analysis.json 的 labels。 + +观察目录因预算省略已有记忆的累计条次为 A 0、B 380（B1 107、B2 122、B3 151），这是重复省略条次，不是不同记忆数。本轮保留既有目录、证据、准入与投影预算；仅将这一现象记录为诊断，不据此作因果归因或放宽规则。 + +调用量与耗时。实际调用来自封存 run.json 的 cases[].model_calls，与运行器 usage 一致，全部 provider_recorded=true；不含连接检查或六名外部判分员。JSON 修正计入实际调用。 + +| 用途／成本（三次合计） | A | B | B/A | +| --- | --- | --- | --- | +| 观察 | 435 | 870 | 2.000 | +| 准入检查 | 395 | 642 | 1.625 | +| JSON 修正 | 5 | 14 | 2.800 | +| 全部实际 GLM 调用 | 835 | 1526 | 1.828 | +| 请求秒数 | 13079.05 | 31124.47 | 2.380 | +| 墙钟分钟 | 222.14 | 524.26 | 2.360 | +| 输入 token | 1519583 | 3099948 | 2.040 | +| 输出 token（服务商记录） | 892287 | 1680753 | 1.884 | + +| 单次请求秒数 | A | B | +| --- | --- | --- | +| p50_seconds | 12.871 | 17.324 | +| p95_seconds | 34.613 | 41.935 | +| max_seconds | 125.898 | 147.908 | + +六次运行墙钟合计 12.44 小时。A 先于 B 串行执行是预登记安排，耗时差包含不同运行时段及实际请求材料差异，不能全部归因为开关自身。各用途及逐次分位数见 timing.json；这只是既有日志统计，不是 CPU 压测。 + +作废与完整性 + +作废运行 0，补跑 0；未发生 429 或未恢复超时。2361 条实际模型调用均记录为 success；导出 infrastructure 的 failed_calls 全部为 0。来源原话字面匹配为 1539/1539，这项真实性检查不替代语义有依据判分。评测中没有修改代码、提示词、冻结语料、评分或虚拟环境；HEAD、语料和三个外部辅助脚本均通过末次完整性核对，工作区干净，test-models.toml 仍被忽略。未提交、推送或合并新的代码。 + +监控期间两次只读状态查询的自动权限审查超时，且一次普通沙箱 SQLite 读取暂时少计；均仅影响进度监控，正式进程继续运行。末次成本全部改用原运行器封存的完整导出核对，无运行因此作废。 + +材料位置 + +/Users/cassia/Local/Code/iris-eval-artifacts/EF-28-EXP2-20261020 + +| 材料 | 相对路径 | +| --- | --- | +| 冻结输入与预登记 | entry_profile_v2.json、spec.json、controller-freeze.json、connection-check.json | +| 全部原始运行、数据库、请求与输出 | raw/A/initial/run-0001..0003、raw/B/initial/run-0001..0003 | +| 三次有效运行的完整序列描述 | groups/A/series.json、groups/B/series.json（指向原始运行） | +| 中性判分材料及各单判员原始结果 | blinded/sample-*/、judgments/sample-*/ | +| 私有对照表 | private-mapping.json | +| 原运行器汇总 | A-summary.json、B-summary.json、paired.json | +| 阶段、逐标签和耗时诊断 | analysis.json、timing.json | +| 代码、语料和隔离方式核对 | integrity.json | +| 本报告 | EXP2-report.md | diff --git a/evals/entry_profile_eval/memory_runner.py b/evals/entry_profile_eval/memory_runner.py index 3059c4f2..bd083748 100644 --- a/evals/entry_profile_eval/memory_runner.py +++ b/evals/entry_profile_eval/memory_runner.py @@ -665,25 +665,221 @@ def _record(self,*args,**kwargs): return manifest +def _sealed(value, field): + return {**value,field:evidence_hash(value)} + + +def _check_seal(value, field): + if value.get(field)!=evidence_hash({k:v for k,v in value.items() if k!=field}): + raise ValueError('summary/series fingerprint mismatch') + + +def _repeat_usage(root): + run=_read(Path(root)/'run.json') + calls=[call for case in run['cases'] for call in case.get('model_calls',[])] + recorded=bool(calls) + if not calls: + # Fake gateways do not write provider call rows. Durable work traces + # retain attempts without counting the same work again at each checkpoint. + for case in run['cases']: + unique={w['id']:w for w in case.get('work',[])} + calls.extend(call for work in unique.values() for call in json.loads(work['calls_json'])) + return dict(calls=len(calls),provider_recorded=recorded, + observe_calls=sum('observe' in c['purpose'] and 'repair' not in c['purpose'] for c in calls), + admit_calls=sum('admit' in c['purpose'] and 'repair' not in c['purpose'] for c in calls), + repair_calls=sum('repair' in c['purpose'] for c in calls), + request_seconds=sum(c.get('duration_ms') or 0 for c in calls)/1000, + prompt_tokens=sum(c.get('prompt_tokens') or c.get('usage',{}).get('prompt_tokens') or 0 for c in calls), + completion_tokens=sum(c.get('completion_tokens') or c.get('usage',{}).get('completion_tokens') or 0 for c in calls)) + + +def run_repeated(corpus, out, *, runs, gateway_factory=None, configs=None): + """Independent complete replays, no automatic retry or selection of cases.""" + import time + from .runner import _source_hash, _model_metadata + if type(runs) is not int or runs<1: raise ValueError('runs must be a positive integer') + if (gateway_factory is None)==(configs is None): raise ValueError('provide exactly one gateway/configuration') + corpus=Path(corpus);root=Path(out);root.mkdir(parents=True,exist_ok=False) + metadata=dict(corpus_sha256=hashlib.sha256(corpus.read_bytes()).hexdigest(),source_sha256=_source_hash(), + scoring_sha256=hashlib.sha256((HERE/'scoring_v3.md').read_bytes()).hexdigest(), + replay_mode='daily_plus_final_flush',models=_model_metadata(configs or {})) + result=dict(format_version=1,evaluation='entry_profile_repeated',metadata=metadata, + requested_runs=runs,settings={},runs=[],invalidated=None,gate_eligible=False) + def save(): _write(root/'series.json',_sealed(result,'series_sha256')) + save() + for index in range(1,runs+1): + if (hashlib.sha256(corpus.read_bytes()).hexdigest()!=metadata['corpus_sha256'] or _source_hash()!=metadata['source_sha256']): + result['invalidated']=dict(run_index=index,reason='input_changed');save();break + directory=f'run-{index:04d}';started=time.monotonic() + try: + manifest=run_sqlite(corpus,root/directory,gateway_factory=gateway_factory,configs=configs) + except Exception as error: + result['invalidated']=dict(run_index=index,reason='runner_error',error_type=type(error).__name__);save();raise + result['runs'].append(dict(index=index,directory=directory,materials_sha256=manifest['materials_sha256'], + gate_eligible=manifest['metadata']['gate_eligible'],invalidated=manifest['metadata']['invalidated'], + usage=_repeat_usage(root/directory),wall_seconds=time.monotonic()-started)) + if manifest['metadata']['invalidated']: + result['invalidated']=dict(run_index=index,**manifest['metadata']['invalidated']);save();break + if (hashlib.sha256(corpus.read_bytes()).hexdigest()!=metadata['corpus_sha256'] + or _source_hash()!=metadata['source_sha256'] + or any(manifest['metadata'][k]!=metadata[k] for k in ('corpus_sha256','source_sha256','scoring_sha256','replay_mode','models'))): + result['invalidated']=dict(run_index=index,reason='input_changed');save();break + result['gate_eligible']=len(result['runs'])==runs and all(r['gate_eligible'] for r in result['runs']) + save() + return _sealed(result,'series_sha256') + + +def _repeat_inputs(root, record): + materials=root/record['directory']/'judging-materials';manifest=_read(materials/'manifest.json') + if (manifest.get('materials_sha256')!=record['materials_sha256'] + or manifest['materials_sha256']!=evidence_hash({k:v for k,v in manifest.items() if k!='materials_sha256'}) + or hashlib.sha256((root/record['directory']/'run.json').read_bytes()).hexdigest()!=manifest['run_sha256'] + or hashlib.sha256((materials/'scoring.md').read_bytes()).hexdigest()!=manifest['metadata']['scoring_sha256']): + raise ValueError('run material fingerprint mismatch') + values=[] + for index,case in enumerate(manifest['cases'],1): + if case['file']!=f'cases/{index:04d}.json': raise ValueError('invalid case filename') + path=materials/case['file'] + if not path.resolve().is_relative_to(materials.resolve()): raise ValueError('material outside directory') + doc=_read(path) + if evidence_hash(doc)!=case['sha256'] or doc['case_id']!=case['case_id']: raise ValueError('case fingerprint mismatch') + values.append((case['case_id'],doc['input'])) + if len({key for key,_ in values})!=len(values): raise ValueError('duplicate case ID') + return manifest,values + + +def _case_statistics(case_id, payload, vote): + rows=list(zip(payload['memories'],vote['memory_results'],strict=True)) + model=[v for m,v in rows if m['author']!='admin'] + return dict(case_id=case_id,label_sha256=evidence_hash(dict(must_cover=payload['must_cover'],forbidden=payload['forbidden'])), + coverage=sum(vote['must_cover']),coverage_total=len(vote['must_cover']), + model_items=len(model),grounded_items=sum(v['supported'] for v in model), + violation_memory_items=sum(bool(v['violations']) for _,v in rows), + model_violation_memory_items=sum(bool(v['violations']) for v in model), + violation_occurrences=sum(len(v['violations']) for _,v in rows)+len(vote['projection_violations']), + projection_violations=len(vote['projection_violations']),forbidden_occurrences=sum(vote['forbidden']), + stage_coverage={key:sum(vote['stage_coverage'][key]) for key in STAGES}) + + +_REPEAT_TOTALS=('coverage','coverage_total','model_items','grounded_items','violation_memory_items', + 'model_violation_memory_items','violation_occurrences','projection_violations','forbidden_occurrences') + + +def summarize_repeated(series, scores): + """Describe existing independent scores; never grade or change vote merging.""" + from statistics import mean + root=Path(series);description=_read(root/'series.json');_check_seal(description,'series_sha256') + if (description['invalidated'] or len(description['runs'])!=description['requested_runs'] + or len(scores)!=description['requested_runs']): + raise ValueError('incomplete/invalid series or missing scores; no runs may be omitted') + runs=[];case_order=None;labels={} + for index,(record,score_path) in enumerate(zip(description['runs'],scores,strict=True),1): + if record['index']!=index or record['directory']!=f'run-{index:04d}' or record['invalidated']: + raise ValueError('invalid repeated run order') + manifest,inputs=_repeat_inputs(root,record);score=_read(Path(score_path)) + if (score['materials_sha256']!=manifest['materials_sha256'] or score['scoring_status']!=SCORING_STATUS + or score['judge_runs'] not in (1,2) or len(score['judgments'])!=len(inputs)): + raise ValueError('score fingerprint/format mismatch') + if any(manifest['metadata'][k]!=description['metadata'][k] for k in ('corpus_sha256','source_sha256','scoring_sha256','replay_mode','models')): + raise ValueError('repeated inputs changed') + cases=[_case_statistics(key,payload,validate_vote(vote,payload)) + for (key,payload),vote in zip(inputs,score['judgments'],strict=True)] + order=[c['case_id'] for c in cases] + if case_order is not None and case_order!=order: raise ValueError('case pairing mismatch') + case_order=order + for case in cases: + if case['case_id'] in labels and labels[case['case_id']]!=case['label_sha256']: raise ValueError('case labels changed') + labels[case['case_id']]=case['label_sha256'] + totals={key:sum(c[key] for c in cases) for key in _REPEAT_TOTALS} + if (totals['coverage']!=score['metrics']['must_cover_covered'] or totals['coverage_total']!=score['metrics']['must_cover_total'] + or totals['model_items']!=score['metrics']['model_items'] or totals['grounded_items']!=score['metrics']['grounded_items'] + or totals['forbidden_occurrences']!=score['metrics']['forbidden_occurrences']): + raise ValueError('score metrics disagree with votes') + totals['grounded_ratio']=totals['grounded_items']/totals['model_items'] if totals['model_items'] else None + runs.append(dict(index=index,**totals,usage=_repeat_usage(root/record['directory']), + wall_seconds=record['wall_seconds'],judge_model=score['judge_model'],judge_runs=score['judge_runs'], + gate_eligible=score['gate_eligible'],cases=cases)) + means={key:mean(r[key] for r in runs) for key in _REPEAT_TOTALS} + means.update(calls=mean(r['usage']['calls'] for r in runs),request_seconds=mean(r['usage']['request_seconds'] for r in runs), + wall_seconds=mean(r['wall_seconds'] for r in runs), + grounded_ratio=mean(r['grounded_ratio'] for r in runs) if all(r['grounded_ratio'] is not None for r in runs) else None) + per_case=[] + for offset,key in enumerate(case_order or []): + values=[r['cases'][offset]['coverage'] for r in runs] + per_case.append(dict(case_id=key,label_sha256=labels[key],coverage_counts=values,coverage_mean=mean(values), + coverage_min=min(values),coverage_max=max(values),coverage_total=runs[0]['cases'][offset]['coverage_total'])) + return _sealed(dict(format_version=1,evaluation='entry_profile_repeated_summary',metadata=description['metadata'], + settings=description['settings'],runs=runs,means=means,per_case=per_case,descriptive_only=True,gate_eligible=False),'summary_sha256') + + +def compare_repeated(left, right): + """Paired by run ordinal and stable case ID, with no selection of good runs.""" + from statistics import mean + for value in (left,right): _check_seal(value,'summary_sha256') + for key in ('corpus_sha256','scoring_sha256','source_sha256','replay_mode','models'): + if left['metadata'][key]!=right['metadata'][key]: raise ValueError('paired '+key+' mismatch') + if not left['runs'] or len(left['runs'])!=len(right['runs']): raise ValueError('paired run count mismatch') + pairs=[];run_pairs=[] + for a,b in zip(left['runs'],right['runs'],strict=True): + if a['index']!=b['index']: raise ValueError('paired run index mismatch') + ac={c['case_id']:c for c in a['cases']};bc={c['case_id']:c for c in b['cases']} + if ac.keys()!=bc.keys(): raise ValueError('paired case mismatch') + for key,x in ac.items(): + y=bc[key] + if x['label_sha256']!=y['label_sha256']: raise ValueError('paired labels mismatch') + pairs.append(dict(index=a['index'],case_id=key,left_coverage=x['coverage'],right_coverage=y['coverage'], + coverage_delta=y['coverage']-x['coverage'], + left_violation_memory_items=x['violation_memory_items'],right_violation_memory_items=y['violation_memory_items'], + left_forbidden_occurrences=x['forbidden_occurrences'],right_forbidden_occurrences=y['forbidden_occurrences'])) + run_pairs.append(dict(index=a['index'],coverage_delta=b['coverage']-a['coverage'], + calls_delta=b['usage']['calls']-a['usage']['calls'],request_seconds_delta=b['usage']['request_seconds']-a['usage']['request_seconds'], + wall_seconds_delta=b['wall_seconds']-a['wall_seconds'])) + per_case=[] + for key in dict.fromkeys(p['case_id'] for p in pairs): + values=[p['coverage_delta'] for p in pairs if p['case_id']==key] + per_case.append(dict(case_id=key,coverage_deltas=values,coverage_delta_mean=mean(values))) + totals={name:{**{key:sum(r[key] for r in value['runs']) for key in _REPEAT_TOTALS}, + 'calls':sum(r['usage']['calls'] for r in value['runs']), + 'request_seconds':sum(r['usage']['request_seconds'] for r in value['runs']), + 'wall_seconds':sum(r['wall_seconds'] for r in value['runs'])} for name,value in (('left',left),('right',right))} + return dict(format_version=1,evaluation='entry_profile_paired_comparison',descriptive_only=True,gate_eligible=False, + left_summary_sha256=left['summary_sha256'],right_summary_sha256=right['summary_sha256'], + left_settings=left['settings'],right_settings=right['settings'],pairs=pairs,run_pairs=run_pairs,per_case=per_case,totals=totals, + means={key:mean(r[key] for r in run_pairs) for key in ('coverage_delta','calls_delta','request_seconds_delta','wall_seconds_delta')}) + + def main(): import argparse - parser=argparse.ArgumentParser(description='Entry-profile memory execution / offline scoring') + parser=argparse.ArgumentParser(description='Entry-profile memory execution / offline scoring / repeated paired comparison') sub=parser.add_subparsers(dest='command',required=True) run=sub.add_parser('run');run.add_argument('--corpus',required=True);run.add_argument('--out',required=True) + repeat=sub.add_parser('repeat');repeat.add_argument('--corpus',required=True);repeat.add_argument('--out',required=True) + repeat.add_argument('--runs',type=int,required=True) score=sub.add_parser('score');score.add_argument('--materials',required=True) score.add_argument('--judgments',action='append',required=True);score.add_argument('--judge-model',required=True);score.add_argument('--out',required=True) + summary=sub.add_parser('summarize');summary.add_argument('--series',required=True);summary.add_argument('--score',action='append',required=True);summary.add_argument('--out',required=True) + compare=sub.add_parser('compare');compare.add_argument('--left',required=True);compare.add_argument('--right',required=True);compare.add_argument('--out',required=True) args=parser.parse_args() - if args.command=='run': + if args.command in ('run','repeat'): from iris.models import load_test_models - manifest=run_sqlite(args.corpus,args.out,configs=load_test_models()) - print(json.dumps(dict(cases=len(manifest['cases']),gate_eligible=manifest['metadata']['gate_eligible'], - invalidated=manifest['metadata']['invalidated']),ensure_ascii=False)) - if manifest['metadata']['invalidated']: - raise SystemExit(2) - else: + if args.command=='repeat': + result=run_repeated(args.corpus,args.out,runs=args.runs,configs=load_test_models()) + print(json.dumps(dict(runs=len(result['runs']),invalidated=result['invalidated']),ensure_ascii=False)) + if result['invalidated']: raise SystemExit(2) + else: + manifest=run_sqlite(args.corpus,args.out,configs=load_test_models()) + print(json.dumps(dict(cases=len(manifest['cases']),gate_eligible=manifest['metadata']['gate_eligible'], + invalidated=manifest['metadata']['invalidated']),ensure_ascii=False)) + if manifest['metadata']['invalidated']: raise SystemExit(2) + elif args.command=='score': result=score_draft(args.materials,args.judgments,judge_model=args.judge_model) - _write(Path(args.out),result) - print(json.dumps(result['metrics'],ensure_ascii=False)) + _write(Path(args.out),result);print(json.dumps(result['metrics'],ensure_ascii=False)) + elif args.command=='summarize': + result=summarize_repeated(args.series,args.score);_write(Path(args.out),result) + print(json.dumps(result['means'],ensure_ascii=False)) + else: + result=compare_repeated(_read(Path(args.left)),_read(Path(args.right)));_write(Path(args.out),result) + print(json.dumps(result['means'],ensure_ascii=False)) if __name__=='__main__': diff --git a/tests/test_entry_profile_repeat_eval.py b/tests/test_entry_profile_repeat_eval.py new file mode 100644 index 00000000..84a51365 --- /dev/null +++ b/tests/test_entry_profile_repeat_eval.py @@ -0,0 +1,254 @@ +"""Repeated replay and paired descriptive statistics, with fake votes only.""" +import copy +import json +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from evals.entry_profile_eval import memory_runner as runner +from iris.db import dumps +from iris.models import ModelError +from test_entry_profile_incremental import Gateway, propose, source +from test_entry_profile_incremental_eval import corpus + + +def factory(case, store): + def observe(payload): + if payload['profiles']: + return {'observations': [dict(op='confirm', memory_id=payload['profiles'][0]['memory_id'], + source_refs=[source(m) for m in payload['messages']])]} + return propose(payload) + return Gateway(store, observe) + + +def scores(series_dir, coverages, violations=None): + series = json.loads((series_dir / 'series.json').read_text()) + paths = [] + for record, coverage in zip(series['runs'], coverages, strict=True): + root = series_dir / record['directory']; materials = root / 'judging-materials' + manifest = json.loads((materials / 'manifest.json').read_text()) + rounds = [] + for name in ('format-a', 'format-b'): + out = root / name; out.mkdir() + (out / 'manifest.json').write_bytes((materials / 'round-template.json').read_bytes()) + for case in manifest['cases']: + value = json.loads((materials / case['file']).read_text())['input'] + bad = bool(violations and violations[record['index'] - 1]) + vote = dict(memory_results=[dict(index=i, memory_id=m['memory_id'], revision=m['revision'], + supported=not bad, violations=['无依据', '个人评价'] if bad else [], reason='假票只验证汇总结构') + for i, m in enumerate(value['memories'], 1)], projection_violations=[], + must_cover=[coverage] * len(value['must_cover']), forbidden=[bad] * len(value['forbidden']), + stage_coverage={stage: [coverage] * len(value['must_cover']) for stage in runner.STAGES}) + (out / case['judgment_file']).write_text(dumps(vote)) + rounds.append(out) + report = runner.score_draft(materials, rounds, judge_model='fake-format-only') + path = root / 'score.json'; path.write_text(dumps(report)); paths.append(path) + return paths + + +def series(tmp_path, name='series', *, runs=2, corpus_path=None): + path = corpus_path or corpus(tmp_path) + root = tmp_path / name + runner.run_repeated(path, root, runs=runs, gateway_factory=factory) + return root + + +def test_repeat_uses_independent_directories_and_databases(tmp_path): + stores = [] + def record(case, store): + stores.append(store.path) + return factory(case, store) + root = tmp_path / 'series' + result = runner.run_repeated(corpus(tmp_path), root, runs=3, gateway_factory=record) + assert result['requested_runs'] == 3 and len(result['runs']) == 3 + assert len(stores) == len(set(map(str, stores))) == 3 + assert [r['index'] for r in result['runs']] == [1, 2, 3] + assert all((root / r['directory'] / 'judging-materials/manifest.json').exists() for r in result['runs']) + with pytest.raises(FileExistsError): + runner.run_repeated(corpus(tmp_path), root, runs=3, gateway_factory=record) + + +@pytest.mark.parametrize('count', [0, -1, True, 1.5]) +def test_invalid_repeat_count_is_rejected_before_creating_output(tmp_path, count): + out = tmp_path / 'out' + with pytest.raises(ValueError): + runner.run_repeated(corpus(tmp_path), out, runs=count, gateway_factory=factory) + assert not out.exists() + + +def test_repeat_stops_after_invalid_transport_instead_of_silently_rerunning(tmp_path): + calls = [] + def bad(case, store): + calls.append(case) + return Gateway(store, lambda p: (_ for _ in ()).throw(ModelError('retryable', 'late', reason='timeout'))) + result = runner.run_repeated(corpus(tmp_path), tmp_path / 'series', runs=3, gateway_factory=bad) + assert len(calls) == len(result['runs']) == 1 + assert result['invalidated']['reason'] == 'timeout' + assert not result['gate_eligible'] + + +def test_summary_preserves_each_run_and_per_case_coverage_distribution(tmp_path): + root = series(tmp_path) + summary = runner.summarize_repeated(root, scores(root, [True, False])) + assert [r['coverage'] for r in summary['runs']] == [1, 0] + assert summary['means']['coverage'] == .5 + assert summary['per_case'][0]['coverage_counts'] == [1, 0] + assert summary['per_case'][0]['coverage_mean'] == .5 + assert not summary['gate_eligible'] and summary['descriptive_only'] + + +def test_pairing_is_by_run_index_and_case_and_sums_violations_separately(tmp_path): + path = corpus(tmp_path) + value = json.loads(path.read_text()); value['cases'][0]['forbidden'] = ['禁止个人评价'] + path.write_text(dumps(value)) + left_root = series(tmp_path, 'left', corpus_path=path) + right_root = series(tmp_path, 'right', corpus_path=path) + left = runner.summarize_repeated(left_root, scores(left_root, [False, True])) + right = runner.summarize_repeated(right_root, scores(right_root, [True, True], [True, False])) + compared = runner.compare_repeated(left, right) + assert [p['coverage_delta'] for p in compared['pairs']] == [1, 0] + assert compared['means']['coverage_delta'] == .5 + assert compared['per_case'][0]['coverage_deltas'] == [1, 0] + assert compared['totals']['right']['violation_memory_items'] == 1 + assert compared['totals']['right']['forbidden_occurrences'] == 1 + assert compared['totals']['right']['calls'] == compared['totals']['left']['calls'] + assert compared['means']['calls_delta'] == 0 + assert compared['descriptive_only'] and not compared['gate_eligible'] + + +def test_missing_or_foreign_score_is_not_averaged_away(tmp_path): + root = series(tmp_path) + paths = scores(root, [True, False]) + with pytest.raises(ValueError): runner.summarize_repeated(root, paths[:1]) + value = json.loads(paths[1].read_text()); value['materials_sha256'] = '0' * 64 + paths[1].write_text(dumps(value)) + with pytest.raises(ValueError, match='fingerprint'): + runner.summarize_repeated(root, paths) + + +def test_comparison_rejects_different_corpus_or_tampered_summary(tmp_path): + root = series(tmp_path) + summary = runner.summarize_repeated(root, scores(root, [True, False])) + tampered = copy.deepcopy(summary); tampered['runs'][0]['coverage'] = 100 + with pytest.raises(ValueError, match='fingerprint'): runner.compare_repeated(summary, tampered) + other_path = tmp_path / 'other.json' + value = json.loads(corpus(tmp_path).read_text()); value['cases'][0]['must_cover'] = ['其他格式标签'] + other_path.write_text(dumps(value)) + other_root = series(tmp_path, 'other', corpus_path=other_path) + other = runner.summarize_repeated(other_root, scores(other_root, [True, True])) + with pytest.raises(ValueError, match='corpus'): runner.compare_repeated(summary, other) + + +def test_offline_commands_do_not_load_any_model_configuration(tmp_path, monkeypatch): + root = series(tmp_path) + paths = scores(root, [True, False]); output = tmp_path / 'summary.json' + import iris.models + monkeypatch.setattr(iris.models, 'load_test_models', lambda: (_ for _ in ()).throw(AssertionError('config read'))) + monkeypatch.setattr(sys, 'argv', ['memory_runner', 'summarize', '--series', str(root), + '--score', str(paths[0]), '--score', str(paths[1]), '--out', str(output)]) + runner.main() + assert json.loads(output.read_text())['means']['coverage'] == .5 + comparison = tmp_path / 'comparison.json' + monkeypatch.setattr(sys, 'argv', ['memory_runner', 'compare', '--left', str(output), + '--right', str(output), '--out', str(comparison)]) + runner.main() + assert json.loads(comparison.read_text())['means']['coverage_delta'] == 0 + + +def test_series_detects_input_changes_during_the_final_run(tmp_path, monkeypatch): + path = corpus(tmp_path); original = runner.run_sqlite + def mutate(*args, **kwargs): + manifest = original(*args, **kwargs) + path.write_text(path.read_text() + '\n') + return manifest + monkeypatch.setattr(runner, 'run_sqlite', mutate) + result = runner.run_repeated(path, tmp_path / 'series', runs=1, gateway_factory=factory) + assert result['invalidated']['reason'] == 'input_changed' + assert not result['gate_eligible'] + + +def test_pairing_matches_case_ids_even_if_summary_case_order_differs(tmp_path): + path = corpus(tmp_path); value = json.loads(path.read_text()) + second = copy.deepcopy(value['cases'][0]); second['id'] = 'unit-second' + second['must_cover'] = ['第二例格式标签']; value['cases'].append(second) + path.write_text(dumps(value)) + root = series(tmp_path, runs=1, corpus_path=path) + summary = runner.summarize_repeated(root, scores(root, [True])) + reordered = copy.deepcopy(summary) + reordered['runs'][0]['cases'].reverse() + reordered.pop('summary_sha256') + reordered = runner._sealed(reordered, 'summary_sha256') + comparison = runner.compare_repeated(summary, reordered) + assert [r['case_id'] for r in comparison['pairs']] == ['unit-daily', 'unit-second'] + assert all(r['coverage_delta'] == 0 for r in comparison['pairs']) + + +def test_repeat_default_requests_match_single_replay_without_runtime_overrides(tmp_path): + path = corpus(tmp_path) + baseline, repeated = [], [] + def capture(target): + def make(case, store): + gateway = factory(case, store) + original = gateway.chat + def chat(messages, purpose, max_tokens=16000, **kwargs): + target.append(dict(messages=copy.deepcopy(messages), purpose=purpose, + max_tokens=max_tokens, deadline=kwargs.get('_deadline'))) + return original(messages, purpose, max_tokens, **kwargs) + gateway.chat = chat + return gateway + return make + runner.run_sqlite(path, tmp_path / 'single', gateway_factory=capture(baseline)) + result = runner.run_repeated(path, tmp_path / 'repeat', runs=1, + gateway_factory=capture(repeated)) + assert baseline == repeated and len(repeated) == 3 + assert result['settings'] == {} + assert result['runs'][0]['usage']['calls'] == 3 + assert result['runs'][0]['usage']['provider_recorded'] is False + + +@pytest.mark.parametrize('command', ['run', 'repeat']) +def test_cli_has_no_experimental_runtime_switch(tmp_path, monkeypatch, command): + import iris.models + monkeypatch.setattr(iris.models, 'load_test_models', + lambda: (_ for _ in ()).throw(AssertionError('config read'))) + args = ['memory_runner', command, '--corpus', str(corpus(tmp_path)), + '--out', str(tmp_path / 'out'), '--union-nomination'] + if command == 'repeat': + args.extend(['--runs', '3']) + monkeypatch.setattr(sys, 'argv', args) + with pytest.raises(SystemExit) as error: + runner.main() + assert error.value.code == 2 + + +def test_repeat_cli_passes_only_standard_run_arguments(tmp_path, monkeypatch): + import iris.models + configs = object() + called = [] + monkeypatch.setattr(iris.models, 'load_test_models', lambda: configs) + def repeated(corpus_path, output, **kwargs): + called.append((corpus_path, output, kwargs)) + return {'runs': [{}, {}, {}], 'invalidated': None} + monkeypatch.setattr(runner, 'run_repeated', repeated) + path = corpus(tmp_path); out = tmp_path / 'out' + monkeypatch.setattr(sys, 'argv', ['memory_runner', 'repeat', '--corpus', str(path), + '--runs', '3', '--out', str(out)]) + runner.main() + assert called == [(str(path), str(out), {'runs': 3, 'configs': configs})] + + +def test_provider_call_usage_counts_observe_admit_and_json_repair(tmp_path): + (tmp_path / 'run.json').write_text(dumps({'cases': [{'model_calls': [ + {'purpose': 'consolidation_entry_profile_observe', 'duration_ms': 100, + 'prompt_tokens': 2, 'completion_tokens': 3}, + {'purpose': 'consolidation_entry_profile_observe_repair', 'duration_ms': 250, + 'prompt_tokens': 5, 'completion_tokens': 7}, + {'purpose': 'consolidation_entry_profile_admit', 'duration_ms': 400, + 'prompt_tokens': 11, 'completion_tokens': 13}, + ]}]})) + usage = runner._repeat_usage(tmp_path) + assert usage == dict(calls=3, provider_recorded=True, observe_calls=1, + admit_calls=1, repair_calls=1, request_seconds=.75, + prompt_tokens=18, completion_tokens=23)