[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

Models trained with special tokens around biological sequences (NatureLM, domain-tokenizer models: <dna>ACGT</dna>) need those tokens at inference. Without them the tokenizer reads the sequence as plain text and the score drops silently; in one case a DNA classification fell to about chance. The wrapper format belongs to the model, but only the task knows which text is which sequence type. This PR makes Model.extra the contract between them, and identifies preprocess by code, so the wrapping shows up only in inference.

1. Model.extra is the capability contract (sieval/core/models/extra.py)

• A task asks the model for a capability by key; an absent key changes nothing. Core interprets no key and only checks the shape of documented ones, so a typo fails before the run.

• First key: sequence_wrappers: {dna|rna|protein|mol: "<format with {seq}>"}. A template without {seq}, with another field, or under an unknown type is refused when the model is built and by --dry-run.

• sequence_wrapper(extra, type) reads a template. No built-in task injects biological sequences, so none is wired here. A plugin task:

  from sieval.core.models import sequence_wrapper
  
  async def preprocess(self, raw, ctx):
      wrap = sequence_wrapper(self.model.extra, "dna")
      seq = wrap.format(seq=raw["sequence"]) if wrap else raw["sequence"]
      return build_prompt_record([{"role": "user", "content": PROMPT.format(seq=seq)}])

  models:
      naturelm:
          name: org/naturelm
          api_base: http://127.0.0.1:30000/v1
          extra:
              sequence_wrappers:
                  dna: "<dna>{seq}</dna>"

2. preprocess is identified by code, not by rendered prompts. It uses the static walk the grading stages use, computed once at setup.

The rendered prompt is a function of	identified by
the code preprocess reaches, and the grader_args() it reads	preprocess ({digest, modules, args})
the samples, and for few-shot the split exemplars come from	data, and preprocess.splits
the model's capabilities (extra)	inference, whose per-sample requests digest the body as sent

• One benchmark on a wrapping model and on a plain one shares data and preprocess and differs only in inference. Reusing one model's inference for the other is refused, as before.
• Arguments. Only the arguments preprocess reads (as self.<name>, self._<name>, or an attribute stored from one) count. A task that overrides grader_args may keep an argument under another name, so there every argument it returns counts. No built-in task overrides it.
• Splits. When the code reads a split, the one its fewshot_split argument names is digested; without that argument, every split but test. Digesting 20k rows takes about 0.1s.
• Dynamic code. If preprocess reaches code through a run-time name (getattr, eval, importlib), or a function it calls opens a file, the identity covers the task's whole code plus the data files beside the reading module, with fallback recording why. Among built-in tasks only Spider 2.0-lite and QuoteBench take this path. Modules that read a file beside them at import (LiveCodeBench few-shot JSON, NL2SH assets, SciCode, AdvancedIF) add that package's data files without falling back. Markdown counts as data; only AGENTS.md, CLAUDE.md and README.md are skipped.
• Blind spots (documented). The walk is static: a prompt that reads an environment variable, or a file through a reader outside the known set (pd.read_csv), is covered only through the argument naming it.
• Resume. A preprocess change now restarts every sample of the task; before, only the samples whose rendered prompt changed. Each preprocess record is stamped with the key, so an interrupted restart leaves preprocess unknown and each sample's stamp decides.
• Old directories (IDENTITY_SCHEMA 3; RESULT_SCHEMA_VERSION stays 1). Schemas 1 and 2 recorded per-sample rendered-prompt digests. On resume, if data matches and every recorded sample re-renders the prompt it recorded, the block is upgraded in place; otherwise preprocess counts as changed. That costs one local preprocess call per recorded sample, during the upgrade only. I chose this over accepting by inference request digests: those are missing under record_meta: false, and they fold the prompt together with everything else sent, so a mismatch cannot name the stage.
• Dry run. plan_run's stages (feat(cli): sieval run --dry-run reports each task's stage plan #220) and task_resume_plan now compare preprocess instead of leaving it out. A preprocess change gives rerun_from: preprocess; legacy_preprocess: true marks a schema-1 or schema-2 directory, and plan_run carries the verdict of its rendering pass. feat(cli): sieval run starts an engine only for samples left to infer #216's lazy launch reads the same plan.

Breaking change

• meta.json identity.schema is 3. Its preprocess is {digest, modules, args, splits?, fallback?}. Readers must still accept schema 1 (and schema 2, written only by unreleased main builds), whose preprocess/prompt is {digest, samples}. The CHANGELOG now carries one migration entry for schemas 2 and 3, since no release wrote schema 2.
• Each preprocess record's stamp is {data, preprocess} instead of {data, prompt}.
• stage_code() takes args= and its preprocess entry gains args, splits and files. ResumePlan gains legacy_preprocess. recorded_sample_digests() returns a SampleDigests mapping.

Related Issues

Closes #218

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check)
• Unit + integration after rebasing onto main (pytest tests/unit tests/integration -n 8 -m "not stress" --cov --cov-fail-under=95): 9623 passed, 1 skipped; coverage 97.16% (gate 95%).
• Earlier full suite with every CI group: 9559 passed, 94 skipped, 3 known unrelated failures (test_exec_quotebench_anchor.py on macOS; test_no_memory_leak_across_runs and test_benchmark_scenarios under -n 8 only, which pass serially).
• pre-commit run -a, scripts/check_preflight.py and scripts/sync_package_stubs.py --check pass.

New tests:

• tests/unit/core/runners/test_sequence_wrappers.py uses a dual-sequence DNA task and two models, one with sequence_wrappers and one without.
  • data and preprocess are equal; inference and its requests differ, and the request bodies show <dna>…</dna> around both sequences.
  • A wrapper for another type leaves the prompt alone; with the key absent, prompts are byte-identical to a task without the reading code.
  • A resume with the wrapping model over a plain run is refused at inference, in both arun and plan_run; task_resume_plan and plan_run.stages show preprocess: same.
  • A changed few-shot split changes preprocess; with fewshot_split named, only that split is digested and an unread split's rows do not move it.
  • sequence_wrapper and the validation refusals are covered.
• tests/unit/core/tasks/test_run_meta_preprocess.py runs on a plugin package on disk.
  • An edit to preprocess, to a helper it calls, or to a module constant changes the identity; an edit to report or feedback does not.
  • An argument preprocess reads changes it, one it does not read leaves it, and under a grader_args override an argument kept under another name still changes it.
  • A file opened at run time falls back, and editing that file changes the identity; a prompt.md template counts while CLAUDE.md does not; a computed getattr falls back.
• tests/unit/core/runners/test_stage_resume.py:
  • A preprocess argument change restarts all 3 samples; a fast resume calls preprocess zero times.
  • An interrupted restart leaves preprocess unknown, and the next run infers only the samples still stamped with the old key.
  • Schema-2 upgrade: unchanged prompts upgrade in place with one rendering pass, then zero passes; a changed prompt restarts every sample; plan_run reports both.
• tests/integration/resume/test_across_releases.py: the 0.10.0, 0.10.2 and 0.11.0 fixture directories resume into schema 3 with zero restarts, regrades and model calls, and the upgraded preprocess equals a fresh run's. A grader argument change still regrades without restarting; a prompt change restarts all 5 samples.
• tests/unit/cli/test_validation.py: a malformed sequence_wrappers under a model's extra, args.extra or a task's infer_args.extra fails the config check.

Each fix from review was checked by reverting it: the grader_args override, Markdown-as-data and named-split tests each fail without their fix.

Manual

• For all 84 built-in tasks, the preprocess walk was listed. Few-shot tasks pick up splits and only the arguments their prompt reads (HumanEval's timeout and k stay out). The tasks above fall back or add package data files. No built-in task overrides grader_args.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/
• Deleted code verified — no remaining call sites depend on it

If: Breaking Change

• Described what breaks and migration path in Summary
• Existing tests updated to reflect new behavior

🤖 Generated with Claude Code