Skip to content

OUTERLOOP_AUTHOR_EFFORT: set a Claude author's effort level - #598

Merged
renmengye merged 2 commits into
outerloop-science:mainfrom
2016choang:feat/author-effort
Oct 27, 2026
Merged

renmengye merged 2 commits into
outerloop-science:mainfrom
2016choang:feat/author-effort

Conversation

@2016choang

@2016choang 2016choang commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

One optional operator setting that sets the author's effort level on every backend.

OUTERLOOP_AUTHOR_EFFORT: the author's effort level.
Every author harness has a reasoning-effort control, but the kernel never sets one, so every author runs at its harness's default. An operator who wants a cheaper author, or a deliberately less capable one for an experiment, can today only change the model.

  • One list for every backend: low, medium, high, xhigh, max, the levels all three harnesses share. Unset or blank leaves each harness's default and its argv and config unchanged.
  • Claude: --effort <level>. The pinned Claude Code release (2.1.285) has the flag; I checked the pinned binary itself (checksum matches harnesses.toml).
  • Codex: -c model_reasoning_effort="<level>" as a kernel-owned setting after the operator's -c args, so project config cannot override it.
  • Hermes: agent.reasoning_effort in the seeded config.yaml. run_agent.main builds its agent without a reasoning config and never reads that key, so hermes_capture.py also hands the level to the agent (via OUTERLOOP_HERMES_EFFORT), the same parse_reasoning_effort value Hermes's own CLI would pass. Unset, the wrapper changes nothing. On a custom_providers endpoint Hermes sends it as a top-level reasoning_effort and drops the field for the rest of the session if the server rejects it.
  • Only the author reads it. Panel judges and the steward keep their defaults on every backend; the wake path passes it only for the author spec.
  • author_config_error refuses an unknown level for every backend, so start, the tick, overrides, rebind and each wake all reject a typo before any spend.
  • It is read when each session starts, like OUTERLOOP_CODEX_WEB_SEARCH, and is not stored in the run record. A parked run picks up a change at its next wake. The key is added to TICK_ENV_KEYS and the deploy step's allowlist, which test_tick_env_keys_match_the_deploy_allowlist keeps identical.
  • docs/reference/environment.md documents it, including that an author on an endpoint profile gets it too.

Verification

  • New tests:
    • test_harness.py: the Claude flag reaches argv only when set; the setting is parsed and normalized, and an unknown level is refused.
    • test_endpoints.py: each backend's own setting at the process boundary on an endpoint, with and without a level (Claude argv, Codex kernel-owned -c after the operator's, Hermes config.yaml and wrapper channel).
    • test_session_evidence.py: the capture wrapper gives the agent the reasoning config only when a level is set.
    • test_attempt.py: author_config_error checks the level for every backend; build_harness passes it through.
    • test_author_overrides.py: the fresh climb and the dispatched wake both pass the current setting on every backend, and a change between them is picked up, unlike the bound model.
  • Ruff, mypy, and pre-commit are clean.
  • The full suite passes locally apart from 11 tests that fail identically on an unmodified main checkout on my machine (confirmed machine-specific in review).

Compatibility: one new optional operator setting. No run records, ledger, inbox, or cache formats change. Older kernels ignore the setting, and deployments that leave it unset behave exactly as before.

The author's effort level, read when each session starts. Judges, the
steward and other backends ignore it; author_config_error refuses an
unknown level at start, in the tick, and on each wake.
@2016choang
2016choang requested a review from renmengye as a code owner October 26, 2026 18:10
@renmengye renmengye added the outerloop:review re-request the advisory review label Oct 26, 2026
@renmengye

Copy link
Copy Markdown
Member

Thanks Chris, this looks good. The full gate passes on the head commit here (3089 passed, 7 skipped), so the 11 local failures look specific to your machine.

One request before merge: OUTERLOOP_AUTHOR_EFFORT has a backend-neutral name, but a Codex or Hermes author silently ignores it. Both harnesses accept the same level names, so could this PR cover them too?

  • Codex: model_reasoning_effort accepts none, minimal, low, medium, high, xhigh, max and ultra. I checked this list in Codex 0.153; the pin is 0.160. It could go in as -c model_reasoning_effort="<level>" beside the kernel-owned -c settings in CodexHarness.run (harness.py around line 1421), so project config cannot override it.
  • Hermes: agent.reasoning_effort in config.yaml takes the same list at the pinned v2026.9.24. HermesHarness already writes a minimal config.yaml (harness.py around line 1758). Hermes's example config says the setting applies to OpenRouter and Nous Portal providers, so please check what it does on a custom_providers endpoint.
  • One list for every backend: accept the five levels all three harnesses share (low, medium, high, xhigh, max), and have author_config_error check the setting for every backend.
  • Keep the author-only scope: judges and the steward stay at their defaults on every backend.
  • Wording: the docs, the CHANGELOG line and the wake-path condition (wake_backend == "claude") would drop the Claude-only wording.

… agent.reasoning_effort

Review: the setting's name is backend-neutral, so a Codex or Hermes author
honors it too. Codex gets a kernel-owned -c model_reasoning_effort. Hermes
gets agent.reasoning_effort in the seeded config.yaml, and the capture
wrapper hands the level to the agent, since run_agent.main builds its agent
without a reasoning config and never reads that key. One level list and one
check for every backend; judges and the steward keep their defaults.
@2016choang

Copy link
Copy Markdown
Contributor Author

Thanks! Pushed 689e55f to cover every backend.

  • One list, one check: low, medium, high, xhigh, max for all three; author_config_error checks the setting for every backend.
  • Codex: -c model_reasoning_effort="<level>" joins the kernel-owned settings after the operator's -c args, so project config cannot override it.
  • Hermes: agent.reasoning_effort goes into the seeded config.yaml, but at v2026.9.24 that key alone never reaches a request on our path. hermes_capture.py runs run_agent.main through agent/legacy_cli.py, which builds AIAgent without a reasoning_config, and AIAgent init never reads agent.reasoning_effort. Only the CLI, gateway, oneshot and ACP surfaces call resolve_reasoning_config, plus fallback activation and /model. So when the harness sets OUTERLOOP_HERMES_EFFORT, the capture wrapper fills reasoning_config from Hermes's own parse_reasoning_effort, the same value its CLI would pass. Unset, the wrapper changes nothing.
  • custom_providers endpoint: the custom profile sends the level as a top-level reasoning_effort. All five levels are on its OpenAI-compatible wire list, so none are clamped. If the server rejects the field, Hermes's turn recovery drops the reasoning fields for the rest of the session. Note that unset is not "nothing on the wire" there: the custom profile already defaults to medium.
  • Scope: judges and the steward stay at their defaults on every backend. The wake path now gates only on the author spec.
  • The docs and CHANGELOG are backend-neutral. New tests cover each backend's setting at the process boundary on an endpoint, and the wrapper with and without a level.

@renmengye

Copy link
Copy Markdown
Member

Thanks Chris, this covers it. Good catch that the pinned Hermes's run_agent.main never reads agent.reasoning_effort; the opt-in constructor patch in the capture wrapper looks like the least invasive way in.

Review of 689e55f (local Codex review, since the review bot does not run on fork PRs yet): no blocking findings. The full gate passes here (3098 passed, 7 skipped). Judges and the steward never get an effort level, the Codex -c value is valid TOML and comes after any operator -c settings, and unset leaves all three backends unchanged.

One optional follow-up, not needed for merge: no test covers the contained Hermes path. Removing the APPTAINERENV_OUTERLOOP_HERMES_EFFORT line leaves the suite green, and a contained Hermes author would then silently run at its default effort. One assertion on the contained env would cover it.

@renmengye renmengye added outerloop:review re-request the advisory review and removed outerloop:review re-request the advisory review labels Oct 27, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 1 — reviewed head 689e55f9 — reviewer hermes/gpt-5.6-terra.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: no defects found.

@renmengye
renmengye merged commit 3fab8b6 into outerloop-science:main Oct 27, 2026
5 checks passed
@renmengye renmengye mentioned this pull request Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

outerloop:review re-request the advisory review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants