[earakely-scale]

What and why

Some tasks run a verifier that checks whether the run was a fair test at all: whether the scripted events fired, whether colleagues stayed on script, whether the solver stayed out of the harness. Today that verifier records its verdict, but nothing acts on it, so a run that fails the check still reports its quality score.

What changes

• A new option on rubrics_verifier. fail_task_when_not_passing: true fails the step with VerifierNotPassingError when the recorded verdict isn't passing, once the verdict has been written.
  • "Passing" means a score of 1.0.
  • The score is computed exactly as before, with any score_aggregator.
  • The error names the rows that weren't met.
• It must stop the task. The option is refused together with fail_task_on_error: false, because a tolerated failure would let the steps after it run and score the run.
• No re-grading. The scheduler doesn't retry a VerifierNotPassingError under retry_config: grading the same run again could pass it.
• Where the error lives. VerifierNotPassingError sits in agent_env.task_step.context, next to StepContextError, and is exported from agent_env.task_step.

With the option set, and the scoring step depending on the verifier, a run that fails the check ends FAILED before anything scores it. Consumers that already drop failed runs then drop it too:

{"id": "validity", "type": "rubrics_verifier", "fail_task_when_not_passing": true, "...": "..."},
{"id": "final-score", "type": "aggregate_verifiers", "verifier_ids": ["rubric-judge"],
 "depends_on": ["rubric-judge", "validity"]}

Docs. The step reference should document fail_task_when_not_passing: what passing means, the fail_task_on_error requirement, and that the scoring step should depend on the verifier.

How it was tested

Unit tests: tst/unit/task_step/verifiers/test_rubrics_verifier_fail_task_when_not_passing.py (10 tests) covers:

• a passing verdict;
• an unmet row, with the verdict kept on the context;
• a partial weighted score;
• a penalized negative row;
• a run the judge couldn't grade;
• the option off;
• the refusal with fail_task_on_error: false;
• storing the option only when it's set;
• a task that fails at the verifier and never runs its scoring step;
• retry_config not grading the run again: three gradings without the scheduler change, one with it.

tst/unit/task_step, tst/unit/task, tst/unit/bundle and tst/unit/runner pass: 4,038 passed, 11 skipped.

End to end: in progress; results will be added here.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no actionable issues were found.

Summary

Adds an opt-in gate that fails a task after a rubric verifier records a score below 1.0.

• Rubric verifiers can fail tasks when their score is below 1.0.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Grade run] --> B[Record verdict]
  B --> C{Gate enabled and score below 1?}
  C -- No --> D[Complete verifier]
  D --> E[Release dependent steps]
  C -- Yes --> F[Raise VerifierNotPassingError]
  F --> G[Skip scheduler retry]
  G --> H[Save failed task and verdict]

Loading

_{Reviews (1) · Last reviewed commit: "feat(rubrics_verifier): fail_task_when_n..." · Reviewed by Greptile}