Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ Versions follow [SemVer](https://semver.org).

## [Unreleased]

- `OUTERLOOP_AUTHOR_EFFORT` sets the author's effort level (`low`, `medium`, `high`, `xhigh` or `max`) on every backend: Claude Code's `--effort` (the pinned 2.1.285 has the flag), Codex's `model_reasoning_effort` as a kernel-owned `-c` setting, and Hermes's `agent.reasoning_effort`. For Hermes the setting is written to the seeded `config.yaml`, and the capture wrapper also hands it to the agent, because `run_agent.main` builds its agent without a reasoning config and would otherwise ignore it. Unset keeps each harness's default. Panel judges and the steward ignore it, so a cheaper author does not change how its work is reviewed. Start, the tick and each wake refuse an unknown level. The setting is read when each session starts and is not stored in run records.
- Run endings append one `attest` trajectory event: an HMAC-SHA256 over the run id and chain head with a deployment key created owner-only beside the operator settings file, and `outerloop trajectory --verify` reports `attested` or `attestation mismatch`. Older kernels ignore the event; no ledger or PR-body change.
- The PR body gains a report-only Number check that lists values in the author's report matching no `measure` event or Experiments result when rounded to the report's decimals; nothing is blocked and no trajectory event is added.
- Session end records each tool call from the retained Claude or Codex log as a `tool.call` trajectory event, once per call even when a session resumes. `outerloop trajectory` and a new Process section in the PR body show counts by tool. Older kernels ignore these events.
Expand Down
17 changes: 17 additions & 0 deletions docs/reference/environment.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ over the inherited environment even when it is empty.
| [`OUTERLOOP_IDENTITY`](#outerloop_identity) | `personal` when the kernel posts as you | unset |
| [`OUTERLOOP_AUTHOR_BACKEND`](#outerloop_author_backend) | the author's harness | `claude` |
| [`OUTERLOOP_AUTHOR_MODEL`](#outerloop_author_model) | the author's model | unset |
| [`OUTERLOOP_AUTHOR_EFFORT`](#outerloop_author_effort) | the author's effort level | unset |
| [`OUTERLOOP_CLAUDE_MODEL`](#outerloop_claude_model) | model for every Claude role | unset |
| [`OUTERLOOP_AUTHOR_ENDPOINT`](#outerloop_author_endpoint) | endpoint profile for the author | unset |
| [`OUTERLOOP_AUTHOR_OVERRIDES`](#outerloop_author_overrides) | per-target author selection | unset |
Expand Down Expand Up @@ -305,6 +306,22 @@ only. A Hermes author needs it, in your provider's model ID.

- Default: unset

### `OUTERLOOP_AUTHOR_EFFORT`

The effort level the author runs at, on every backend: Claude Code's
`--effort`, Codex's `model_reasoning_effort`, or Hermes's
`agent.reasoning_effort`. Lower levels spend fewer tokens per turn. Panel
judges and the steward ignore it. An author on an endpoint profile gets it
too, so set it there only when the server accepts the backend's effort
parameter. Hermes is the exception: it sends a top-level `reasoning_effort`
field and drops the field for the rest of the session if the server rejects
it. Start, the tick and each wake refuse a value outside the list. Unlike
the author's model, it is not stored in the run's record, so a parked run
picks up a change at its next wake.

- Values: `low`, `medium`, `high`, `xhigh` or `max`
- Default: unset, which keeps each harness's own default

### `OUTERLOOP_CLAUDE_MODEL`

The model for every Claude role: the author when its backend is claude,
Expand Down
3 changes: 2 additions & 1 deletion scripts/tick_deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,8 @@ if [ -n "$ENV_TRUSTED" ]; then
OUTERLOOP_HERMES_REF OUTERLOOP_HERMES_SHA \
OUTERLOOP_CACHE_ROOT REVIEW_BACKEND OUTERLOOP_BRIDGE_RUNTIME \
OUTERLOOP_AUTHOR_ENDPOINT REVIEW_ENDPOINT REVIEW_MODEL \
OUTERLOOP_AUTHOR_BACKEND OUTERLOOP_AUTHOR_MODEL OUTERLOOP_AUTHOR_OVERRIDES \
OUTERLOOP_AUTHOR_BACKEND OUTERLOOP_AUTHOR_MODEL OUTERLOOP_AUTHOR_EFFORT \
OUTERLOOP_AUTHOR_OVERRIDES \
OUTERLOOP_CLAUDE_MODEL \
OUTERLOOP_CLAUDE_BIN OUTERLOOP_CODEX_BIN OUTERLOOP_CODEX_KEY_FILE \
OUTERLOOP_HERMES_KEY_FILE OUTERLOOP_HERMES_RESUME_MAX_CHARS \
Expand Down
7 changes: 7 additions & 0 deletions src/outerloop/attempt.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@
ClaudeModelUnset,
Harness,
ResumeContextBlocked,
author_effort,
default_binary,
default_claude_model,
redact,
Expand Down Expand Up @@ -258,6 +259,10 @@ def author_config_error(
env = os.environ if environ is None else environ
if backend not in ("claude", "codex", "hermes"):
return f"unknown author backend {backend!r} (expected 'claude', 'codex' or 'hermes')"
try:
author_effort(env)
except ValueError as exc:
return str(exc)
if backend == "hermes":
from outerloop.harness import hermes_resume_max_chars
from outerloop.hermes_install import hermes_ready
Expand Down Expand Up @@ -6047,6 +6052,7 @@ def _run_id(value: str) -> str:
hermes_repo=Path(os.environ["REVIEW_HERMES_REPO"])
if wake_backend == "hermes"
else None,
effort=author_effort() if role_spec is author_spec else "",
)
wake_secrets = tuple(k for k in (*forge.secrets(), *wake_panel_secrets, wake_api_key) if k)
if end_requested(Path(args.run_root), args.resume):
Expand Down Expand Up @@ -6199,6 +6205,7 @@ def _run_id(value: str) -> str:
hermes_repo=Path(os.environ["REVIEW_HERMES_REPO"])
if args.author_backend == "hermes"
else None,
effort=author_effort(),
),
spec=spec,
panel_lenses=panel_lenses,
Expand Down
1 change: 1 addition & 0 deletions src/outerloop/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@
"REVIEW_MODEL",
"OUTERLOOP_AUTHOR_BACKEND",
"OUTERLOOP_AUTHOR_MODEL",
"OUTERLOOP_AUTHOR_EFFORT",
"OUTERLOOP_AUTHOR_OVERRIDES",
"OUTERLOOP_CLAUDE_MODEL",
"OUTERLOOP_CLAUDE_BIN",
Expand Down
39 changes: 39 additions & 0 deletions src/outerloop/harness.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,25 @@ def default_claude_model() -> str:
return model


# The effort levels all three author harnesses accept: Claude Code's
# `--effort`, Codex's `model_reasoning_effort` and Hermes's
# `agent.reasoning_effort`.
EFFORT_LEVELS = ("low", "medium", "high", "xhigh", "max")


def author_effort(env: Mapping[str, str] | None = None) -> str:
"""The author's effort level from OUTERLOOP_AUTHOR_EFFORT, for any backend.
Unset or blank keeps the harness's own default. Raises ValueError for a
value outside EFFORT_LEVELS. Judges and the steward never read it."""
value = (os.environ if env is None else env).get("OUTERLOOP_AUTHOR_EFFORT", "")
value = value.strip().lower()
if value and value not in EFFORT_LEVELS:
raise ValueError(
f"OUTERLOOP_AUTHOR_EFFORT must be one of {', '.join(EFFORT_LEVELS)} (got {value!r})"
)
return value


@dataclass(frozen=True)
class SessionResult:
"""What happened in one session — everything the orchestrator needs to
Expand Down Expand Up @@ -657,6 +676,9 @@ class ClaudeCodeHarness:
# ignored when set. Contained sessions get the ADC file bind-mounted.
vertex: VertexConfig | None = None
endpoint: EndpointProfile | None = None
# `--effort` level; "" keeps Claude Code's default. Only the author sets it
# (OUTERLOOP_AUTHOR_EFFORT), so judges keep their default depth.
effort: str = ""

CONTAINER_CLAUDE = "/opt/agent/claude"
CONTAINER_ADC = "/opt/agent/adc.json"
Expand Down Expand Up @@ -706,6 +728,7 @@ def run(
"--permission-mode",
permission_mode,
*([("--safe-mode" if oauth else "--bare")] if self.bare else []),
*(["--effort", self.effort] if self.effort else []),
*self.extra_args,
]
if resume_session_id:
Expand Down Expand Up @@ -1173,6 +1196,9 @@ class CodexHarness:
# (the deployment's runner is the boundary)
container_image: str = ""
apptainer_binary: str = field(default_factory=apptainer_from_env)
# `model_reasoning_effort`; "" keeps Codex's default. Only the author sets
# it (OUTERLOOP_AUTHOR_EFFORT).
effort: str = ""
# in-container path the host codex binary is bound to (bind-from-host, like
# ClaudeCodeHarness.CONTAINER_CLAUDE — the image stays codex-free, and codex
# is updated by swapping one host binary, no rebuild). A bare class attribute
Expand Down Expand Up @@ -1399,6 +1425,8 @@ def run(
kernel_owned: tuple[str, ...] = ("-c", "features.multi_agent=false")
if not codex_web_search(bool(self.endpoint)):
kernel_owned += ("-c", 'web_search="disabled"')
if self.effort:
kernel_owned += ("-c", f"model_reasoning_effort={json.dumps(self.effort)}")
codex_argv = _codex_command(
self.CONTAINER_CODEX if self.container_image else self.binary,
self.model,
Expand Down Expand Up @@ -1682,6 +1710,9 @@ class HermesHarness:
apptainer_binary: str = field(default_factory=apptainer_from_env)

resume_max_chars: int | None = None
# `agent.reasoning_effort`; "" keeps Hermes's default. Only the author sets
# it (OUTERLOOP_AUTHOR_EFFORT).
effort: str = ""

@capture_session("hermes")
def run(
Expand Down Expand Up @@ -1749,6 +1780,8 @@ def run(
f" key_env: {json.dumps(self.key_env)}\n"
" api_mode: chat_completions\n"
)
if self.effort:
config_lines.append(f"agent:\n reasoning_effort: {json.dumps(self.effort)}\n")
if self.approvals_deny:
config_lines.append("approvals:\n deny:\n")
config_lines += [f' - "{glob}"\n' for glob in self.approvals_deny]
Expand Down Expand Up @@ -1812,12 +1845,18 @@ def run(
env = session_env(self.api_key, self.key_env, session_home)
env["TERMINAL_CWD"] = str(workspace.resolve())
env["OUTERLOOP_CAPTURE_ID"] = capture_id
if self.effort:
# run_agent.main builds its agent without the config's
# reasoning_effort; hermes_capture applies it (see there)
env["OUTERLOOP_HERMES_EFFORT"] = self.effort
if self.container_image:
# --cleanenv drops the host env except APPTAINERENV_*: the key
# travels via the environment, never argv
env[f"APPTAINERENV_{self.key_env}"] = env[self.key_env]
env["APPTAINERENV_TERMINAL_CWD"] = env["TERMINAL_CWD"]
env["APPTAINERENV_OUTERLOOP_CAPTURE_ID"] = capture_id
if self.effort:
env["APPTAINERENV_OUTERLOOP_HERMES_EFFORT"] = self.effort
# cwd is the per-run home, NOT the workspace: --save_sample writes
# its trajectory JSON to cwd, and artifacts must never land in the
# clone (they would enter the diff).
Expand Down
21 changes: 20 additions & 1 deletion src/outerloop/hermes_capture.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
"""Run the pinned Hermes CLI unchanged, retaining its reported session evidence.
"""Run the pinned Hermes CLI, retaining its reported session evidence.

The one behavior change is the author's effort level. `run_agent.main` builds
its agent without a reasoning config, so the `agent.reasoning_effort` the
harness writes to config.yaml would never reach a request. When the harness
sets OUTERLOOP_HERMES_EFFORT, the agent gets the reasoning config Hermes's own
CLI would build from that level. Unset, nothing changes.

This file runs with Hermes's Python, without an outerloop installation.
"""
Expand All @@ -18,6 +24,19 @@ def main() -> None:
import run_agent
from agent.legacy_cli import main as cli_main

if effort := os.environ.get("OUTERLOOP_HERMES_EFFORT", ""):
from hermes_constants import parse_reasoning_effort

reasoning = parse_reasoning_effort(effort)
original_init = run_agent.AIAgent.__init__

def init(agent: Any, *args: Any, **kwargs: Any) -> None:
if kwargs.get("reasoning_config") is None:
kwargs["reasoning_config"] = reasoning
original_init(agent, *args, **kwargs)

run_agent.AIAgent.__init__ = init

original = run_agent.AIAgent.run_conversation
root_agent: Any = None

Expand Down
8 changes: 7 additions & 1 deletion src/outerloop/role_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ def build_harness(
hermes_repo: Path | None = None,
hermes_provider: str = "",
endpoint: str = "",
effort: str = "",
) -> Harness:
"""Construct the harness for any role on any backend — the ONE deployment
wiring (`spec.tools` → native flags, `spec.budget` → turns/walltime,
Expand All @@ -118,7 +119,9 @@ def build_harness(
Containment is NOT decided here: pass `container_image` where the
deployment has a jail (the cluster), pass none where the runner itself is
the ephemeral boundary (CI). The tokenless split keeps credentials out of
the session either way."""
the session either way. `effort` is each backend's own reasoning-effort
setting (`--effort`, `model_reasoning_effort`, `agent.reasoning_effort`);
callers pass it for the author only."""
model, profile = resolve_endpoint(model or "", backend, endpoint)
if profile:
api_key = profile.key()
Expand All @@ -135,6 +138,7 @@ def build_harness(
timeout_s=spec.budget.walltime_s,
container_image=container_image,
extra_args=(*codex_extra_args, *web),
effort=effort,
)
if backend == "hermes":
if hermes_repo is None:
Expand Down Expand Up @@ -165,6 +169,7 @@ def build_harness(
enabled_toolsets=enabled,
disabled_toolsets=tuple(t for t in _HERMES_TOOLSETS if t not in enabled),
container_image=container_image,
effort=effort,
)
if backend != "claude":
raise ValueError(f"unknown backend: {backend!r}")
Expand All @@ -185,6 +190,7 @@ def build_harness(
# role on every CLI flips together and the API key stays the fallback
vertex=None if profile else vertex_from_env(),
endpoint=profile,
effort=effort,
)


Expand Down
15 changes: 15 additions & 0 deletions tests/test_attempt.py
Original file line number Diff line number Diff line change
Expand Up @@ -1114,6 +1114,18 @@ def test_codex_author_config_error() -> None:
assert "unknown author backend" in codex_author_config_error("typo", "m", "img.sif")


@pytest.mark.parametrize(
"backend, model, image", [("claude", "claude-x", ""), ("codex", "gpt-5.6-terra", "img.sif")]
)
def test_author_config_error_checks_the_effort(backend: str, model: str, image: str) -> None:
from outerloop.attempt import author_config_error

bad = {"OUTERLOOP_AUTHOR_EFFORT": "minimal"} # Codex knows it; not every backend does
assert "OUTERLOOP_AUTHOR_EFFORT" in author_config_error(backend, model, image, environ=bad)
good = {"OUTERLOOP_AUTHOR_EFFORT": "low"}
assert author_config_error(backend, model, image, environ=good) == ""


def test_resolve_author_key_file(monkeypatch, tmp_path) -> None:
"""Per-backend author keys COEXIST and are selected by backend; an explicit
path wins, else the per-backend env var, else the default file."""
Expand Down Expand Up @@ -1890,6 +1902,9 @@ def test_harness_is_built_from_the_spec() -> None:
assert harness.timeout_s == 120
assert harness.container_image == "img.sif"
assert harness.bare is False # an editor keeps the target repo's guidance
assert harness.effort == "" # Claude Code's default unless the author sets one
lowered = build_harness("sk-key", spec, effort="low")
assert isinstance(lowered, ClaudeCodeHarness) and lowered.effort == "low"

spec = author_spec(max_turns=9, walltime_s=300)
harness = build_harness(
Expand Down
5 changes: 5 additions & 0 deletions tests/test_author_overrides.py
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,7 @@ def test_override_credential_separation(deployment, same_path):
@pytest.mark.parametrize("queued", [False, True])
@pytest.mark.parametrize("long_session", [False, True])
def test_binding_survives_setting_change(deployment, monkeypatch, backend, queued, long_session):
monkeypatch.setenv("OUTERLOOP_AUTHOR_EFFORT", "low")
monkeypatch.setenv(
"OUTERLOOP_AUTHOR_OVERRIDES",
json.dumps(
Expand Down Expand Up @@ -262,6 +263,7 @@ def launch(**kw):
)
assert attempt.main() == 0
assert (seen["backend"], seen["model"]) == (backend, "served-model[endpoint=onprem]")
assert seen["effort"] == "low"
record = load_record(deployment.run_root, seen["record"].run_id)
assert record.author_overridden
if long_session:
Expand Down Expand Up @@ -297,8 +299,11 @@ def launch(**kw):
],
)
seen.pop("backend")
# effort is read per session, never bound to the run like the model
monkeypatch.setenv("OUTERLOOP_AUTHOR_EFFORT", "high")
assert attempt.main() == 0
assert (seen["backend"], seen["model"]) == (backend, "served-model[endpoint=onprem]")
assert seen["effort"] == "high"

if long_session:
assert seen["spec"].budget.walltime_s == 180 * 60
Expand Down
51 changes: 51 additions & 0 deletions tests/test_endpoints.py
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,57 @@ def communicate(self, **kwargs: Any) -> tuple[str, str]:
assert "endpoint-secret" not in seen.get("config", "")


@pytest.mark.parametrize("backend", ["claude", "codex", "hermes"])
@pytest.mark.parametrize("effort", ["", "low"])
def test_author_effort_reaches_each_backend(
profile: dict[str, str],
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
backend: str,
effort: str,
) -> None:
seen: dict[str, Any] = {}
workspace = tmp_path / "workspace"
workspace.mkdir()
monkeypatch.setattr(harness_mod, "hermes_ready", lambda _: True)

class Process:
returncode = 0

def __init__(self, command: list[str], **kwargs: Any):
seen.update(argv=command, env=kwargs["env"])
path = Path(kwargs["env"]["HOME"]) / ".hermes/config.yaml"
if path.is_file():
seen["config"] = yaml.safe_load(path.read_text())

def communicate(self, **kwargs: Any) -> tuple[str, str]:
return '{"type":"result","subtype":"success","result":"done"}', ""

monkeypatch.setattr(harness_mod.subprocess, "Popen", Process)
harness = build_harness(
"ignored-key",
author_spec(),
backend=backend,
endpoint="local",
binary="/opt/agent-cli",
hermes_repo=tmp_path / "hermes",
effort=effort,
)
harness.run("brief", workspace)
argv, env = seen["argv"], seen["env"]
if backend == "claude":
assert (argv[argv.index("--effort") + 1] == "low") if effort else "--effort" not in argv
elif backend == "codex":
setting = 'model_reasoning_effort="low"'
assert (setting in argv) is bool(effort)
if effort: # kernel-owned: after the operator's own -c settings
assert argv.index(setting) > argv.index("features.multi_agent=false")
else:
agent = seen["config"].get("agent", {})
assert agent.get("reasoning_effort") == (effort or None)
assert env.get("OUTERLOOP_HERMES_EFFORT") == (effort or None)


@pytest.mark.parametrize("backend", ["claude", "codex", "hermes"])
def test_reviewer_endpoint(
profile: dict[str, str], monkeypatch: pytest.MonkeyPatch, backend: str
Expand Down
Loading
Loading