[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

• A key repeated within one mapping of the eval config is now an error on every entry point. sieval run, sieval eval, both --dry-runs and EvalSession(...) read the config through one loader, sieval/cli/_config_yaml.py::load_eval_config. It raises DuplicateKeyError (a ValueError) listing every repeat: <path>: duplicate key 'api_base' at line 5 (first at line 4).
• Before: --dry-run only warned. run, eval and EvalSession used bare yaml.safe_load and silently kept the last value, so a second capabilities: was lost before [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25's schema validator saw it.
• run and run --dry-run refuse before any launch planning (checked in _run_all / _run_dry_run, not left to EvalSession, which builds after engines start). Under --dry-run it is a failed yaml_syntax check (exit 1).
• Otherwise safe_load semantics hold. Keys are compared after construction (1 and 0x1 collide). Keys a YAML merge (<<: *anchor) brings in may be overridden, including through a chain of anchors. Each mapping is checked once, in flatten_mapping, on its keys as written.
• Scope: the eval config only. Recipe YAML, sieval infer target files and files sieval wrote itself (read_persisted_config, the effective_config diff) keep yaml.safe_load. test_no_other_yaml_load_in_the_cli pins every yaml.*load under sieval/cli.
• Unknown top-level keys stay a warning: promoting them would refuse configs refactor(cli)!: scope SiEval to an evaluation engine #199 kept running (alignment:). That belongs with the RFC [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25 capability-name errors.
• BREAKING: a config that repeats a key is now refused. Migration: delete the repeat, keeping the last value (the one the run used). sieval.cli.validation.load_yaml_with_duplicate_check is removed (its only caller was the dry-run). In -o json dry-run output, yaml_syntax.warnings is now always empty; a repeat is the check's detail. Not in any identity; resume unaffected; no RESULT_SCHEMA_VERSION bump. The CHANGELOG entry is staged for the release cut.

Related Issues

Closes #255
Refs #25

Test Plan

Automated

• ruff check, ruff format --check, pre-commit hooks: clean
• pdm run ty check (tests/ included): All checks passed
• pytest tests/unit tests/integration -m "not stress": 10055 passed, 1 skipped

New tests/unit/cli/test_config_yaml.py:

• Loader: parses as safe_load; top-level, nested and multiple duplicates; merge overrides allowed (flat, and a chain through a nested anchor); a duplicate after a merge is still refused; keys equal after construction; independent loads.
• Every entry point: CLI run, run --dry-run, eval, eval --dry-run (-o json, exit 1), EvalSession(...), and _run_all / _run_dry_run with _prepare_launch_batch not called.
• Reverse mutation: each of the four call sites swapped back to bare yaml.safe_load fails at least one test. The previous construct_mapping-based check fails the nested-anchor merge test.

tests/unit/cli/test_validation.py: the last-wins helper tests go with the helper; test_duplicate_key_warnings becomes test_duplicate_key_is_an_error.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/ (core/ untouched)
• Deleted code verified — no remaining call sites depend on it (load_yaml_with_duplicate_check had one caller)

If: Breaking Change

• Described what breaks and migration path in Summary
• Existing tests updated to reflect new behavior

🤖 Generated with Claude Code