feat(core)!: Model.extra as a capability contract, and a code-based preprocess identity

AGENTS.md

@@ -15,7 +15,7 @@ SiEval is a **model delivery quality verification system**: eval-side knowledge
 Precise reproducibility is a product contract, not a nicety.
 
 - Safety guards (e.g. the resume stage comparison) ship strict-only. No `--force-*` flags, no bypass env vars, no "(future)" escape-hatch hints in errors.
-- A resume (`--resume`, `--resume DIR`) compares stage identities, not config text (`sieval/core/runners/stage_resume.py`), and reruns from the earliest changed stage: changed data or preprocess → those samples rerun; changed postprocess, feedback or report → rerun from that stage on the recorded inference (report only: recomputed from the records, no engine or judge); changed inference, or an unknown stage it would reuse → refused. Before that, `meta.json` `result_schema_version` must equal `RESULT_SCHEMA_VERSION` (absent: mapped from `version`, 0.10.0+ is 1); the package version is provenance only. `effective_config.yaml` and `infer_plans.yaml` are audit records; config changes go to `meta.json` `runs[].config`.
+- A resume (`--resume`, `--resume DIR`) compares stage identities, not config text (`sieval/core/runners/stage_resume.py`), and reruns from the earliest changed stage: changed data → those samples rerun; changed preprocess (its code and arguments, not the rendered prompts) → every sample reruns; changed postprocess, feedback or report → rerun from that stage on the recorded inference (report only: recomputed from the records, no engine or judge); changed inference, or an unknown stage it would reuse → refused. Before that, `meta.json` `result_schema_version` must equal `RESULT_SCHEMA_VERSION` (absent: mapped from `version`, 0.10.0+ is 1); the package version is provenance only. `effective_config.yaml` and `infer_plans.yaml` are audit records; config changes go to `meta.json` `runs[].config`.
 - Recovery: "start fresh" or "match the stage the refusal names". Escape-hatch proposals must re-justify the contract, not just add a flag.
 
 ## Result Schema Version
@@ -25,7 +25,9 @@ Bump `RESULT_SCHEMA_VERSION` (`sieval/core/tasks/run_meta.py`) in the PR that ma
 - a change to a task result directory's layout (shards, `.idx`, `meta.json`, the `report.json` shape) that older directories cannot be read under;
 - a core change that alters results. The grading identities do not walk `sieval.core` or `sieval.datasets`, so nothing else catches it.
 
-`IDENTITY_SCHEMA` versions only the `identity` block. Bumping it does not refuse a resume, so a new schema must read the old one: schema 2 reads schema 1's `grader` through `legacy_grader_keys`, which keeps the old walks unchanged.
+`IDENTITY_SCHEMA` versions only the `identity` block. Bumping it does not refuse a resume, so a new schema must read the old one: schema 2 reads schema 1's `grader` through `legacy_grader_keys`, which keeps the old walks unchanged; schema 3 reads the rendered-prompt `preprocess` of schemas 1 and 2 by rendering the recorded samples once.
+
+`Model.extra` is the capability contract between tasks and models (`sieval/core/models/extra.py`): a task reads a key in `preprocess`, an absent key changes nothing, and `core/` only checks the shape of documented keys.
 
 ## Project Structure & Module Organization
 

CHANGELOG.md

@@ -11,17 +11,20 @@ Planned as 0.12.0.
 
 ### Migrating from 0.11.x
 
-- **`meta.json` `identity` is schema 2**: `data`, `preprocess` (was `prompt`), `inference`, `postprocess`, `feedback`, `report`. The combined `grader` is gone; judges and services move to `feedback`. Readers must accept schema 1 too. `RESULT_SCHEMA_VERSION` stays 1, so 0.10.x and 0.11.0 directories resume, and an unchanged task's schema-1 directory upgrades in place without regrading.
+- **`meta.json` `identity` is schema 3** (#211, #218): `data`, `preprocess` (was `prompt`), `inference`, `postprocess`, `feedback`, `report`. The combined `grader` is gone; judges and services move to `feedback`. `preprocess` is `{digest, modules, args, splits?, fallback?}`, the code and arguments that render a prompt, instead of a digest of the rendered prompts. Each `preprocess` record is stamped `{data, preprocess}` instead of `{data, prompt}`. Readers must accept schema 1 too (and schema 2, written only by unreleased `main` builds). `RESULT_SCHEMA_VERSION` stays 1, so 0.10.x and 0.11.0 directories resume: an unchanged task's directory upgrades in place without regrading, after one local `preprocess` call per recorded sample.
 - `TaskSaver.begin_run(identity={...})` and `set_identity(**stages)` replace the per-stage keyword arguments.
 
 ### Changed
 
+- **`preprocess` is identified by code, not by the prompts it renders** (#218): the static walk the grading stages use, the `grader_args()` it reads, and for few-shot a digest of the split it draws from. A model's `extra` reaches the request, which `inference` digests, so a wrapping and a plain model share `preprocess` and differ in `inference`. A run-time name or a file opened at run time falls back to the task's whole code and the data files beside the reader. **A preprocess change now restarts every sample of the task**, where before only the samples whose prompt changed restarted.
+
 - **A resume reruns from the earliest stage that changed** (#210). An extraction change reruns from `postprocess`, a scoring change (judge, prompt, sampling, service, `feedback` code) from `feedback`, and a `report`-only change recomputes `report.json` from the records with no engine or judge call. Before, each of these regraded every sample. Code is attributed to stages by a static walk; a run-time lookup (`getattr(self, <expr>)`, `eval`, `importlib`) falls back to all grading code. No built-in task falls back.
 - `runs[].resume` counts what a resume did with each recorded sample.
 - **`sieval run` starts an engine only for a model with samples to infer** (closes #213). A run that may resume (`--resume`, or `resume` in a `runner_config`) first plans each task's resume, read-only, through the runner's own resume planning (`TaskRunner.plan_run`), and launches only the engines whose models some task still has samples to infer with. A resume that reruns only grading stages, or re-reports, starts no engine and runs on a node without GPUs; when some sample needs inference and `nvidia-smi` finds no GPU, the run is refused before anything starts. The plans, `infer_plans.yaml`, the inference identity and its request digest are what an eager launch records. A fresh run is not planned and launches as before. A sample that `feedback` sends to another iteration after a rerun from `feedback` fails at `infer` when its engine was not started, and a resume that starts it retries it.
 
 ### Added
 
+- **`Model.extra` is the capability contract between tasks and models, starting with `sequence_wrappers`** (#218). A task asks for a capability by key and does nothing different when it is absent. `sequence_wrappers: {<seq_type>: "<format with {seq}>"}` (types `dna`, `rna`, `protein`, `mol`) gives the special tokens a model was trained with around biological sequences; a task reads it with `sequence_wrapper(self.model.extra, <type>)` in `preprocess`. A malformed template is refused when the model is built and by `--dry-run`. No built-in task injects such sequences yet.
 - **`SIEVAL_TARGET_GPU` declares the GPUs a serving plan is resolved for**, as `<name>:<count>:<memory MiB>` with the name and memory `nvidia-smi` reports, e.g. `NVIDIA H100 80GB HBM3:8:81559`. Recipe resolution reads it instead of `nvidia-smi`, on any node, so a CPU node resolves the plan, and the inference identity, a node with those GPUs would. Unset, nothing changes. Declared, an engine launches only on a node with exactly those GPUs. `infer_plans.yaml` gains `gpu`, the GPUs its plans were resolved for, outside every identity; a `sieval run --resume` declaring the same GPUs reuses the recorded plan of each served model whose `path`, `infer`, capability layer and `deterministic` are unchanged, unless the plan held an engine secret, which `infer_plans.yaml` records redacted (#215, closes #212).
 - `sieval.core.runners.task_resume_plan` / `plan_resume`: the per-stage resume plan, without running anything.
 - `TaskRunner.plan_run` / `MultiTaskRunner.plan_run` (`RunPlan`): the runner's per-sample plan of a run, `to_infer` included, read-only; the one `sieval run` launches by.

docs/guide/configuration.md

@@ -191,6 +191,49 @@ true, top_logprobs: N}`), each input token's top-N lands in
 the generated tokens'. Served by `sglang_legacy` and `openai_completions`.
 The reply grows by N entries per input token.
 
+## Model Capabilities (`extra`)
+
+`extra` is a dict on a model that tasks read and the API never sees: a task
+asks for a capability by key and does nothing different when it is absent.
+SiEval records it in the inference identity and checks the shape of the keys
+below, so a typo fails before the run. Set it on a model (top level or
+`args`) or per task in `infer_args`; a task's `infer_args.extra` replaces the
+model's.
+
+```yaml
+models:
+    naturelm:
+        name: org/naturelm
+        api_base: http://127.0.0.1:30000/v1
+        extra:
+            sequence_wrappers:
+                dna: "<dna>{seq}</dna>"
+                rna: "<rna>{seq}</rna>"
+```
+
+| Key | Value | Read by |
+| --- | --- | --- |
+| `sequence_wrappers` | `{<seq_type>: "<format with {seq}>"}`; types `dna`, `rna`, `protein`, `mol` (SMILES) | Tasks that put a biological sequence in a prompt |
+
+`sequence_wrappers` is for models trained with special tokens around
+biological sequences (NatureLM, domain-tokenizer models); without them the
+score degrades silently. Each value is a format string whose only field is
+`{seq}` (literal braces as `{{` and `}}`).
+
+A task wraps each sequence it injects in `preprocess`:
+
+```python
+from sieval.core.models import sequence_wrapper
+
+async def preprocess(self, raw, ctx):
+    wrap = sequence_wrapper(self.model.extra, "dna")
+    seq = wrap.format(seq=raw["sequence"]) if wrap else raw["sequence"]
+    return build_prompt_record([{"role": "user", "content": PROMPT.format(seq=seq)}])
+```
+
+With no wrapper the prompt is unchanged. A wrapping model and a plain one
+share `data` and `preprocess` and differ in `inference` (see [Resume](#resume)).
+
 ## Task Pipeline
 
 Each task implements a typed 5-stage pipeline:
@@ -261,7 +304,8 @@ run's, and reruns each sample from the earliest stage that changed:
 
 | Differs | Resume |
 | --- | --- |
-| `data` (the selected samples) or `preprocess` (the prompt) | Those samples rerun from the start; the rest are kept |
+| `data` (the selected samples) | Those samples rerun from the start; the rest are kept |
+| `preprocess` (prompt rendering: its code and the arguments it reads) | Every sample reruns from the start |
 | `inference` | Refused, naming the stage |
 | `postprocess` (answer extraction) | `postprocess`, `feedback` and `report` rerun on the recorded inference |
 | `feedback` (scoring: judges, their prompts and sampling, services) | `feedback` and `report` rerun on the recorded `postprocess` |
@@ -276,15 +320,34 @@ class and `grader_args()`; `feedback` also holds every judge and service.
 those in `GRADER_ARGS_EXCLUDED`; a task extends it with arguments that cannot
 change a verdict: `Task.GRADER_ARGS_EXCLUDED | {"grade_timeout"}`.
 
+The `preprocess` identity is the code `preprocess` reaches, the
+`grader_args()` it reads (all of them when the task overrides `grader_args`)
+and, for few-shot, a digest of the `fewshot_split` it draws from (every split
+but `test` when it names none). It is computed from code at setup, not from
+the prompts: the model's `extra` reaches the request, which `inference`
+digests. A preprocess change restarts every sample of the task.
+
+The walk is static. A prompt that reads an environment variable, or a file
+through a reader it cannot see (`pd.read_csv`), is covered only through the
+argument naming it: keep such inputs in task arguments.
+
 Code is attributed to stages by a static walk from each stage method; code two
 stages reach counts in both. A stage that reaches code by a name only known at
 run time (`getattr(self, name)`, `eval`, `importlib`) counts all grading code,
 and so does every stage after it; `stage_code()` names the construct.
+`preprocess` falls back alone, and also when its code opens a file at run time
+(`open`, `read_text`, `importlib.resources`, a directory listing, a database
+connection): it then counts the task's whole code and the data files beside
+the module that reads one.
 
 A schema-1 `meta.json` (0.10.0 to 0.11.x) folds the grading stages into one
 `grader`. If it still matches the task's code, every verdict is kept and the
 block is upgraded in place; otherwise every sample regrades.
 
+Schemas 1 and 2 recorded `preprocess` as rendered-prompt digests. A resume
+renders each recorded sample once: if all match, the block is upgraded to
+schema 3 in place; otherwise every sample reruns.
+
 A same-directory resume reruns a sample with no recorded identity, such as one
 interrupted before `preprocess`. Under `record_meta: false` no sample has one,
 so every recorded sample reruns, with a warning.
@@ -349,11 +412,12 @@ in `SIEVAL_TARGET_GPU` ([Resolving for declared GPUs](infer.md#resolving-for-dec
       "samples": 3,
       "resume_plan": {"kept": 2, "rereported": 0, "regraded": 0, "repostprocessed": 0, "restarted": 0, "dropped": 0},
       "stages": {
-        "identity_schema": 2,
-        "stages": {"data": "differs", "inference": "same", "postprocess": "same", "feedback": "same", "report": "same"},
+        "identity_schema": 3,
+        "stages": {"data": "differs", "preprocess": "same", "inference": "same", "postprocess": "same", "feedback": "same", "report": "same"},
         "rerun_from": null,
         "refused": null,
-        "legacy_grader": null
+        "legacy_grader": null,
+        "legacy_preprocess": false
       },
       "new": 1,
       "to_infer": 1,
@@ -378,10 +442,12 @@ in `SIEVAL_TARGET_GPU` ([Resolving for declared GPUs](infer.md#resolving-for-dec
   counts the samples sent to the model under test, `to_grade` those run through
   a grading stage; `needs_inference` is whether `to_infer` is above zero.
 - `tasks.<task>.stages` compares each stage's recorded identity with this
-  run's: `same`, `differs` or `unknown`. `rerun_from` is the grading stage the
-  resume reruns from (`report` for a re-report), else `null`; a data change
-  shows in `resume_plan` as `restarted`. `legacy_grader` is `kept` or `regrade`
-  for a schema-1 directory. `stages` is `null` for a fresh start.
+  run's: `same`, `differs` or `unknown`. `rerun_from` is the stage the resume
+  reruns from (`preprocess` restarts every sample, `report` is a re-report),
+  else `null`; a data change shows in `resume_plan` as `restarted`.
+  `legacy_grader` is `kept` or `regrade` for a schema-1 directory;
+  `legacy_preprocess` is `true` for a schema-1 or schema-2 directory, compared
+  by rendering its recorded samples once. `stages` is `null` for a fresh start.
 - `needs_inference` is whether the run starts any engine. A task that infers
   through an `api_base` model counts in its `to_infer` but starts none.
 

sieval/cli/validation.py

@@ -33,6 +33,7 @@
     validate_task_config_args,
 )
 from sieval.cli.session import RootConfigDict
+from sieval.core.models import check_extra
 from sieval.core.models.capabilities import (
     CAPABILITY_KEYS,
     CapabilityConfigError,
@@ -209,6 +210,18 @@ def _validate_models(cfg: dict, result: ValidationResult) -> None:
                 except (TypeError, ValueError) as exc:
                     result.errors.append(str(exc))
 
+        raw_model_args = mcfg.get("args")
+        for where, extra in (
+            (f"Model '{name}' extra", mcfg.get("extra")),
+            (
+                f"Model '{name}' args.extra",
+                raw_model_args.get("extra")
+                if isinstance(raw_model_args, dict)
+                else None,
+            ),
+        ):
+            _check_model_extra(extra, where, result)
+
         if has_base:
             base_ref = mcfg["base"]
             if not isinstance(base_ref, str) or not base_ref:
@@ -523,6 +536,19 @@ def _validate_tasks(cfg: dict, result: ValidationResult) -> None:
                         f"Task '{name}' infer_args cannot change binding resources: "
                         + ", ".join(misplaced)
                     )
+                _check_model_extra(
+                    infer_args.get("extra"), f"Task '{name}' infer_args.extra", result
+                )
+
+
+def _check_model_extra(extra: object, where: str, result: ValidationResult) -> None:
+    """Report an ``extra`` whose documented keys are malformed
+    (:func:`~sieval.core.models.check_extra`), such as a wrapper without
+    ``{seq}``."""
+    try:
+        check_extra(extra, where)
+    except ValueError as exc:
+        result.errors.append(str(exc))
 
 
 def _validate_runner_config(cfg: dict, result: ValidationResult) -> None:

sieval/core/models/__init__.py

@@ -50,6 +50,7 @@
     get_dialect_spec,
 )
 from .exceptions import CapabilityError
+from .extra import SEQUENCE_TYPES, SEQUENCE_WRAPPERS, check_extra, sequence_wrapper
 from .gen_model import GenModel
 from .ir import (
     FINISH_SOURCE_CANDIDATE,
@@ -145,6 +146,8 @@
     "FINISH_SOURCE_CANDIDATE",
     "REQUEST_ENVELOPE_HTTP",
     "REQUEST_ENVELOPE_KEY",
+    "SEQUENCE_TYPES",
+    "SEQUENCE_WRAPPERS",
     "FINISH_SOURCE_PROMPT_BLOCK",
     "FunctionToolCall",
     "GenModel",
@@ -198,6 +201,8 @@
     "TopKEntry",
     "Transport",
     "UsageStats",
+    "check_extra",
+    "sequence_wrapper",
     "ASYNC_HTTP_JSON_FACTORY",
     "AsyncHTTPJSONConnection",
     "BindingReconcileInput",