[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

• What: the inference identity now records numerics, the values the served model computes with: effective dtype, runtime quantization, kv_cache_dtype, and model overrides (json_model_override_args / hf_overrides). A resume refuses when one of these differs. tp/dp/ep/pp, replicas, disaggregation, memory fraction, chunked prefill, CUDA graph, scheduler and ports stay in runs[].serving and are never compared.
• Why: the same weights served in fp8, at another dtype or with kv_cache_dtype: fp8_e5m2 gives different logits. Until now such a run resumed into a bf16 run of the same checkpoint and compared equal to it.
• One list: sieval.core.tasks.run_meta.NUMERICS_PARAMS maps each numerics key to the engine params that set it. numerics_key(param) tells an orchestrator whether one of its overrides changes the identity.
• Effective values, not stated ones: an unset or auto dtype resolves to the checkpoint's torch_dtype (unknown for float32, which engines downcast by version), an unset quantization to its quantization_config.quant_method (None for a readable unquantized checkpoint, so runtime quantization: fp8 on a bf16 checkpoint is a change; unknown only when config.json is unreadable), and an unset kv_cache_dtype to auto. Aliases (bf16, half, fp8 for KV) are normalized, and override JSON is compared as an object. Stating a default is therefore no change. The checkpoint is read in the infer layer (checkpoint_numerics) and passed down as plain data, so core/ stays free of upper-layer imports.
• Endpoints: for an api_base model, the root model may state serving.numerics (validated against the same keys). A key it does not state is unknown. Numerics are stored next to the digest rather than inside it, and a key unknown on either side never counts as a mismatch.
• Older results: IDENTITY_SCHEMA goes from 4 to 5. A schema-4 directory's numerics are read from the plan of its latest own run in runs[].serving, with unset values resolved against this run's checkpoint. A recorded bf16 run therefore resumes against a new bf16 run and refuses an fp16 one. Schema-4 endpoint runs have no plan, and a latest own run that recorded no serving is not backfilled from an older run, so in both cases the numerics are unknown and never block. Schemas 1–3 still compare the whole recipe, as before.
• Refusal message: when the digests match but numerics differ, the refusal names each key, e.g. the served model's numerics: kv_cache_dtype: 'auto' -> 'fp8_e5m2'.

Breaking / migration:

• A resume that changes dtype, quantization, kv_cache_dtype or model overrides is now refused (before, it was silently accepted). Migration: start fresh, or match the recorded values.
• Roles with different values record each one (a sorted list), so a prefill/decode plan whose roles disagree is compared on all of them.
• No RESULT_SCHEMA_VERSION bump: the change is additive, and older directories still read and resume.

Related Issues

Closes #258. Refs #232, #252.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check, pre-commit hooks pass)
• Type check clean (bare ty check --python <venv>, which covers tests/ too): All checks passed
• Unit + integration: pytest tests/unit tests/integration -n 16 -m "not stress" → 10091 passed, 1 skipped
• scripts/sync_package_stubs.py --check clean

New tests:

• test_run_meta.py: which params count, effective defaults and aliases, checkpoint-derived quantization (an unquantized checkpoint is None, not unknown), float32 under auto is unknown, unknown keys, override JSON, roles that disagree, endpoint-stated numerics, reading a schema-4 directory's numerics from its plan (skipping another model's run; unknown when the latest own run recorded no serving), and read_identity on schema 4.
• test_runner_run_meta.py (end to end through TaskRunner): another kv_cache_dtype refuses and leaves meta.json unchanged; runtime quantization: fp8 on an unquantized checkpoint refuses; a stated default plus another tp resumes; a schema-4 directory resumes on matching effective values and refuses fp16; an endpoint that states nothing resumes; an endpoint stating another dtype or quantization refuses.
• test_stage_resume.py: plan_resume / dry-run verdicts.
• test_introspect.py, test_validation.py, test_session.py: the checkpoint reader, serving.numerics validation, and session wiring.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring (existing modules, already marked)
• No new upper-layer dependencies added to core/
• Deleted code verified — no remaining call sites depend on it (none deleted)

If: Breaking Change

• Described what breaks and migration path in Summary
• Existing tests updated to reflect new behavior (schema pins 4→5; the recipe-not-digested test now asserts the numerics beside the digest)

🤖 Generated with Claude Code