feat(core)!: count the served model's numerics in the inference identity

AGENTS.md

@@ -25,7 +25,7 @@ Bump `RESULT_SCHEMA_VERSION` (`sieval/core/tasks/run_meta.py`) in the PR that ma
 - a change to a task result directory's layout (shards, `.idx`, `meta.json`, the `report.json` shape) that older directories cannot be read under;
 - a core change that alters results. The grading identities do not walk `sieval.core` or `sieval.datasets`, so nothing else catches it.
 
-`IDENTITY_SCHEMA` versions only the `identity` block. Bumping it does not refuse a resume, so a new schema must read the old one: schema 2 reads schema 1's `grader` through `legacy_grader_keys`, which keeps the old walks unchanged; schema 3 reads the rendered-prompt `preprocess` of schemas 1 and 2 by rendering the recorded samples once.
+`IDENTITY_SCHEMA` versions only the `identity` block. Bumping it does not refuse a resume, so a new schema must read the old one: schema 2 reads schema 1's `grader` through `legacy_grader_keys`, which keeps the old walks unchanged; schema 3 reads the rendered-prompt `preprocess` of schemas 1 and 2 by rendering the recorded samples once; schema 5 reads schema 4's missing `inference.numerics` from the plan in its `runs[].serving`.
 
 `Model.extra` is the capability contract between tasks and models (`sieval/core/models/extra.py`): a task reads a key in `preprocess`, an absent key changes nothing, and `core/` only checks the shape of documented keys.
 

docs/guide/configuration.md

@@ -364,7 +364,7 @@ run's, and reruns each sample from the earliest stage that changed:
 | --- | --- |
 | `data` (the selected samples) | Those samples rerun from the start; the rest are kept |
 | `preprocess` (prompt rendering: its code and the arguments it reads) | Every sample reruns from the start |
-| `inference` (what the model is asked: request arguments, dialect, the task's inference arguments, `deterministic`, `max_iterations`, `max_retries`, context limit) | Refused, naming the stage |
+| `inference` (what the model is asked: request arguments, dialect, the task's inference arguments, `deterministic`, `max_iterations`, `max_retries`, context limit; and what the served model computes with: `dtype`, `quantization`, `kv_cache_dtype`, model overrides) | Refused, naming the stage |
 | `postprocess` (answer extraction) | `postprocess`, `feedback` and `report` rerun on the recorded inference |
 | `feedback` (scoring: judges, their prompts and sampling, services) | `feedback` and `report` rerun on the recorded `postprocess` |
 | `report` (metric aggregation) | `report.json` is recomputed from the recorded samples: no engine starts, no judge is called |
@@ -379,6 +379,22 @@ entry records its own `serving` (`engine`, `engine_version`, `launch`:
 `in_run` or `external`, and `recipe`: the plan without checkpoint or host), so
 a reader sees what changed.
 
+What the served model computes with does count: these engine parameters
+change the logits, so they are compared as `identity.inference.numerics`.
+
+| `numerics` key | Engine parameters | Unset means |
+| --- | --- | --- |
+| `dtype` | `dtype` | The checkpoint's `torch_dtype`; unknown for float32, which engines downcast |
+| `quantization` | `quantization` | The checkpoint's `quantization_config.quant_method`, else none |
+| `kv_cache_dtype` | `kv_cache_dtype` | `auto` |
+| `model_overrides` | `json_model_override_args` (SGLang), `hf_overrides` (vLLM) | None |
+
+Values are compared as the engine uses them, so stating a default (`dtype:
+auto`, `bf16`) is no change. A key unknown on either side is no difference:
+an endpoint's numerics are only what its root model states under
+`serving.numerics`. A schema-4 `meta.json` has its numerics read from its
+latest run's plan.
+
 The context limit does count, since it decides which prompts overflow. A
 launched engine's is the plan's `max_model_len` (vLLM) or `context_length`
 (SGLang), the smallest across roles; a plan with a role that pins neither
@@ -396,6 +412,7 @@ models:
       engine_version: 0.5.4
       topology: {prefill: 1, decode: 3}
       hardware: 4x H200
+      numerics: {dtype: bfloat16, kv_cache_dtype: fp8_e4m3}   # compared
 ```
 
 - `weights`: a resume compares the model under test by it when both runs
@@ -404,7 +421,7 @@ models:
 - `serving`: `engine_version`, `topology` and `hardware`, any value; never
   compared. Refused on a model `sieval run` launches, which records its own.
   So a single-node run can resume on a prefill/decode deployment of the same
-  weights.
+  weights. `numerics` takes the keys of the table above and is compared.
 
 None of the three is allowed on a derived model.
 
@@ -514,7 +531,7 @@ in `SIEVAL_TARGET_GPU` ([Resolving for declared GPUs](infer.md#resolving-for-dec
       "samples": 3,
       "resume_plan": {"kept": 2, "rereported": 0, "regraded": 0, "repostprocessed": 0, "restarted": 0, "dropped": 0},
       "stages": {
-        "identity_schema": 4,
+        "identity_schema": 5,
         "stages": {"data": "differs", "preprocess": "same", "inference": "same", "postprocess": "same", "feedback": "same", "report": "same"},
         "rerun_from": null,
         "refused": null,

sieval/cli/session.py

@@ -149,6 +149,7 @@
 from sieval.core.utils.paths import resolve_data_dir
 from sieval.datasets.downloaders.staged import probe_source, staged_dir
 from sieval.infer import deployment_plan_projection
+from sieval.infer.introspect import checkpoint_numerics
 from sieval.infer.params import merge_params
 from sieval.infer.topology.models import DETERMINISTIC_DEFAULT_SEED
 
@@ -4880,6 +4881,9 @@ def _setup_tasks(self) -> None:
                 }
                 or None,
                 stated_serving=self._stated_serving_for(candidate.config_name),
+                checkpoint_numerics=self._checkpoint_numerics_for(
+                    candidate.config_name
+                ),
                 resume_config_diff=list(self._resume_config_diff),
                 legacy_match=self._legacy_match_for(task_name, runner_config),
                 unserved_candidate=(
@@ -5087,6 +5091,18 @@ def _judge_weights_for(self, task_name: str) -> dict[str, str] | None:
                 out[role] = weights
         return out or None
 
+    def _checkpoint_numerics_for(self, model_name: str) -> dict[str, JSONValue] | None:
+        """:func:`checkpoint_numerics` of the checkpoint *model_name*'s root
+        launches; ``None`` when SiEval does not serve it."""
+        plan = (self._infer_plans or {}).get(self._root_model(model_name))
+        if plan is None:
+            return None
+        checkpoint = plan.get("checkpoint")
+        if not isinstance(checkpoint, str):
+            return None
+        numerics = checkpoint_numerics(checkpoint)
+        return dict(numerics) if numerics is not None else None
+
     def _stated_serving_for(self, model_name: str) -> dict[str, JSONValue] | None:
         """The root model's ``serving``; ``None`` when absent or empty."""
         root = self._root_model(model_name)

sieval/cli/validation.py

@@ -56,7 +56,7 @@
 from sieval.core.runners.barriers import check_barrier_stage
 from sieval.core.tasks.consts import TaskAction
 from sieval.core.tasks.python_requires import MissingPythonRequirementsError
-from sieval.core.tasks.run_meta import STATED_SERVING_KEYS
+from sieval.core.tasks.run_meta import NUMERICS_PARAMS, STATED_SERVING_KEYS
 
 # Known top-level keys from RootConfigDict
 _ROOT_KEYS: set[str] = set(RootConfigDict.__annotations__)
@@ -166,6 +166,16 @@ def _check_stated_serving(
     version = stated.get("engine_version")
     if version is not None and not _is_nonempty_str(version):
         result.errors.append(f"{where}.engine_version must be a non-empty string")
+    numerics = stated.get("numerics")
+    if numerics is not None and not isinstance(numerics, Mapping):
+        result.errors.append(f"{where}.numerics must be a mapping")
+    elif numerics is not None:
+        unknown = sorted(set(numerics) - set(NUMERICS_PARAMS))
+        if unknown:
+            result.errors.append(
+                f"{where}.numerics has unknown key(s) {', '.join(unknown)}; "
+                f"expected {', '.join(NUMERICS_PARAMS)}"
+            )
 
 
 def _validate_models(cfg: dict, result: ValidationResult) -> None:

sieval/core/runners/multi_runner.py

@@ -138,6 +138,7 @@ def add_task(
         judge_weights: dict[str, str] | None = None,
         lifecycles: dict[str, dict[str, str]] | None = None,
         stated_serving: dict[str, JSONValue] | None = None,
+        checkpoint_numerics: dict[str, JSONValue] | None = None,
         resume_config_diff: list[str] | None = None,
         legacy_match: LegacyStageMatch | None = None,
         unserved_candidate: bool = False,
@@ -147,9 +148,9 @@ def add_task(
         """Register a task for parallel execution.
 
         *inference_recipe*, *context_limit*, *weights*, *judge_weights*,
-        *lifecycles*, *stated_serving*, *serving_group*, *resume_config_diff*,
-        *legacy_match*, *unserved_candidate* and *engine_guard* pass through to
-        :class:`TaskRunner`.
+        *lifecycles*, *stated_serving*, *checkpoint_numerics*, *serving_group*,
+        *resume_config_diff*, *legacy_match*, *unserved_candidate* and
+        *engine_guard* pass through to :class:`TaskRunner`.
 
         Auto-sets ``result_dir`` to ``<base_result_dir>/<task.name>`` when
         a base was provided and the task's config lacks its own.
@@ -193,6 +194,7 @@ def add_task(
             judge_weights=judge_weights,
             lifecycles=lifecycles,
             stated_serving=stated_serving,
+            checkpoint_numerics=checkpoint_numerics,
             resume_config_diff=resume_config_diff,
             legacy_match=legacy_match,
             unserved_candidate=unserved_candidate,

sieval/core/runners/runner.py

@@ -76,6 +76,7 @@
     GradingIdentity,
     InferenceIdentity,
     JudgeIdentity,
+    Numerics,
     PreprocessIdentity,
     ResumeOutcomes,
     RunFrom,
@@ -94,9 +95,12 @@
     judge_identities,
     legacy_grader_keys,
     legacy_inference_digest,
+    numerics_differences,
     preprocess_key,
     prompt_of,
+    read_recorded_numerics,
     request_digest,
+    served_numerics,
     serving_fingerprint,
     stage_code,
     stage_key,
@@ -522,6 +526,7 @@ def __init__(
         judge_weights: dict[str, str] | None = None,
         lifecycles: dict[str, dict[str, str]] | None = None,
         stated_serving: dict[str, JSONValue] | None = None,
+        checkpoint_numerics: dict[str, JSONValue] | None = None,
         resume_config_diff: list[str] | None = None,
         legacy_match: LegacyStageMatch | None = None,
         unserved_candidate: bool = False,
@@ -544,7 +549,8 @@ def __init__(
         that needs a lost engine (:mod:`.engine_loss`). *judge_weights* names,
         by judge role, the weights a launched judge serves, compared instead of
         its name. *lifecycles* are the launched models' lifecycles, recorded in
-        ``runs[].serving``.
+        ``runs[].serving``. *checkpoint_numerics* is the launched
+        checkpoint's own ``dtype`` and ``quantization`` (:func:`served_numerics`).
         """
         # A malformed judge reference would leave every grader identity
         # unknown; refuse it before any work.
@@ -557,6 +563,7 @@ def __init__(
         self._judge_weights = dict(judge_weights or {})
         self._lifecycles = lifecycles
         self._stated_serving = stated_serving
+        self._checkpoint_numerics = checkpoint_numerics
         self._resume_config_diff = list(resume_config_diff or [])
         self._legacy_match = LegacyStageMatch(**(legacy_match or {}))
         self._unserved_candidate = unserved_candidate
@@ -2253,11 +2260,15 @@ def _check_inference(self, current: InferenceIdentity | None) -> None:
             if self._inference_matches(current):
                 return
             recorded = identity["inference"].get("digest")
-            reason = (
-                "this run's inference identity is unknown"
-                if current is None
-                else f"recorded {recorded}, now {current['digest']}"
+            numerics = numerics_differences(
+                self._recorded_numerics(), (current or {}).get("numerics")
             )
+            if current is None:
+                reason = "this run's inference identity is unknown"
+            elif recorded == current["digest"] and numerics:
+                reason = "the served model's numerics: " + "; ".join(numerics)
+            else:
+                reason = f"recorded {recorded}, now {current['digest']}"
         elif identity is None and (
             self._trusts_unrecorded() or self._legacy_match.get("inference") is True
         ):
@@ -2272,7 +2283,9 @@ def _check_inference(self, current: InferenceIdentity | None) -> None:
             f"  recorded: {self._recorded_dir()}\n"
             "  Inference is reused, never rerun, by a resume. Start fresh in a "
             "new result_dir, or match the model's request arguments, the "
-            "task's inference arguments and the context limit.",
+            "task's inference arguments, the context limit and the served "
+            "model's numerics (dtype, quantization, kv_cache_dtype, model "
+            "overrides).",
             task=self._task.name,
         )
 
@@ -2368,6 +2381,7 @@ def _stage_plan(self) -> ResumePlan:
             current,
             self._legacy_grader_keys(),
             self._legacy_inference_digest(),
+            self._checkpoint_numerics,
         )
         if plan["legacy_preprocess"] and self._legacy_prompts_match is not None:
             plan["stages"]["preprocess"] = (
@@ -2832,6 +2846,11 @@ def _inference_identity(self) -> InferenceIdentity | None:
                 task_args=self._task.inference_args(),
                 runner_keys=self._runner_keys(),
                 context_limit=self._context_limit,
+                numerics=served_numerics(
+                    self._inference_recipe,
+                    self._stated_serving,
+                    self._checkpoint_numerics,
+                ),
             )
         except (TypeError, ValueError, AttributeError) as exc:
             logger.warning("Inference identity unavailable: {}", exc)
@@ -2862,8 +2881,14 @@ def _inference_matches(self, current: InferenceIdentity | None) -> bool | None:
             (self._recorded or {}).get("identity"),
             current["digest"] if current is not None else None,
             self._legacy_inference_digest(),
+            numerics=current.get("numerics") if current is not None else None,
+            recorded_numerics=self._recorded_numerics(),
         )
 
+    def _recorded_numerics(self) -> Numerics | None:
+        """The recorded run's :func:`read_recorded_numerics`."""
+        return read_recorded_numerics(self._recorded, self._checkpoint_numerics)
+
     def _serving(self) -> ServingFingerprint | None:
         try:
             return serving_fingerprint(
@@ -3113,7 +3138,7 @@ async def _end_run(
         if inference is not None:
             await self._saver.set_identity(
                 inference={
-                    "digest": inference["digest"],
+                    **inference,
                     "requests": self._requests_identity(),
                 }
             )

sieval/core/runners/stage_resume.py

@@ -63,6 +63,8 @@
     preprocess_key,
     prompt_of,
     read_identity,
+    read_recorded_numerics,
+    served_numerics,
     split_digests,
     stage_code,
     stage_key,
@@ -194,6 +196,7 @@ def plan_resume(
     current: Mapping[str, Any],
     legacy_grader_keys: Iterable[str] = (),
     legacy_inference: str | None = None,
+    checkpoint_numerics: Mapping[str, JSONValue] | None = None,
 ) -> ResumePlan:
     """Compare a recorded ``meta.json`` with this run's identities, running nothing.
 
@@ -203,8 +206,9 @@ def plan_resume(
     :func:`~sieval.core.tasks.run_meta.legacy_grader_keys` for this task,
     read only for a schema-1 directory, and *legacy_inference*
     :func:`~sieval.core.tasks.run_meta.legacy_inference_digest`, only for one
-    before identity schema 4. The runner plans per sample and is
-    authoritative; this says what a resume of a finished directory does.
+    before identity schema 4; *checkpoint_numerics* resolves a schema-4 plan's
+    unset numerics. The runner plans per sample and is authoritative; this
+    says what a resume of a finished directory does.
     """
     block = (recorded_meta or {}).get("identity")
     schema = block.get("schema") if isinstance(block, Mapping) else None
@@ -221,6 +225,8 @@ def plan_resume(
         block if isinstance(block, Mapping) else None,
         mine.get("digest") if isinstance(mine, Mapping) else None,
         legacy_inference,
+        numerics=mine.get("numerics") if isinstance(mine, Mapping) else None,
+        recorded_numerics=read_recorded_numerics(recorded_meta, checkpoint_numerics),
     )
     stages["inference"] = "unknown" if match is None else _verdict(match)
     stages = {stage: stages[stage] for stage in STAGES}
@@ -285,16 +291,17 @@ def task_resume_plan(
     runner_keys: Mapping[str, JSONValue],
     context_limit: int | None = None,
     recipe: Mapping[str, JSONValue] | None = None,
+    stated_serving: Mapping[str, JSONValue] | None = None,
+    checkpoint_numerics: Mapping[str, JSONValue] | None = None,
 ) -> ResumePlan:
     """:func:`plan_resume` for *task* against a recorded ``meta.json``.
 
     Computes this run's inference and grading identities as
     :class:`~sieval.core.runners.runner.TaskRunner` does, so call it after
     ``task.setup()``, which may resolve an argument the grading identities
     read. *runner_keys* holds :data:`~sieval.core.tasks.run_meta.INFERENCE_RUNNER_KEYS`
-    and *context_limit* and *recipe* as the runner is given them; *recipe* is
-    read only for a directory before identity schema 4. Reads files,
-    starts nothing and calls no model.
+    and the rest as the runner is given them. Reads files, starts nothing and
+    calls no model.
     """
     task_cls = type(task)
     judges = judge_identities(task.judge_models(), task.judge_prompts())
@@ -311,6 +318,7 @@ def task_resume_plan(
             task_args=task.inference_args(),
             runner_keys=runner_keys,
             context_limit=context_limit,
+            numerics=served_numerics(recipe, stated_serving, checkpoint_numerics),
         ),
         **grading,
     }
@@ -324,6 +332,7 @@ def task_resume_plan(
             runner_keys=runner_keys,
             recipe=recipe,
         ),
+        checkpoint_numerics,
     )