feat: let a build configure its outer optimizer trial's harbor flags

[shehabyasser-scale]

What broke

The tau3 optimizer trial dies at teardown, after codex has already finished optimizing. From the 2026-07-25 run (runs/tau3-rerun/jobs/2026-07-25__07-33-29, ~40 min, $1.57 burned for zero signal):

"n_errored_trials": 1,
"n_retries": 0,
"exception_stats": {"StreamTerminatedError": ["task__cmDWxgt"]}

task__cmDWxgt/exception.txt puts it in harbor's finalization verifier, not in the model:

harbor/trial/single_step.py:105  _run_verifier
harbor/verifier/verifier.py:199  await self.environment.exec(...)
harbor/environments/dind_compose.py:121  _compose_exec
harbor/environments/modal.py:1304  _sdk_exec  ->  grpclib StreamTerminatedError: Connection lost

Harbor prints the remedy itself, as line 4 of that very job's log:

DinD mode uses host networking: no port isolation between services, no Docker DNS service discovery (extra_hosts entries map service names to 127.0.0.1 instead), and no network namespace isolation. Use --ek modal_vm_runtime=true to use the VM runtime instead which does not use host networking.

DinD funnels the whole sandbox through one host-networked gRPC stream, so a single drop takes down a 20+ minute trial with no partial credit.

Why the previous version of this PR was wrong

It raised tau3's max_retries 1 → 3. That knob is the nested per-case retry: build/config.py → HarborBackendConfig.max_retries → the inner harbor run --max-retries in backend.py::_harbor_command. The failing layer is the outer trial, which never gets --max-retries at all. The live rerun confirmed it: still retries=0, still "Not retrying trial because the maximum number of retries has been reached". Reverted to 1.

Same trap applies to extra_harbor_args: it also only reaches the nested eval. There was no way for a build to say anything about the environment the optimizer itself runs in.

The fix

Add optimizer_harbor_args to HarborBuildConfig, forwarded by vero harbor run to the outer harbor run, and set tau3's to ["--ek", "modal_vm_runtime=true"].

• Forwarded before any trailing CLI args, so the command line still wins (harbor's parse_kwargs at cli/utils.py:65 is last-value-wins per key).
• Validator rejects the four flags the CLI owns (-p, -a, -e, -m), mirroring the existing extra_harbor_args guard.
• The compiled compose addresses services by name (http://eval-sidecar:8000, http://inference-gateway:8001), which the VM runtime resolves through real Docker DNS rather than the DinD extra_hosts → 127.0.0.1 remap, so this is the better-supported path for this task shape, not just the more resilient one.

Live verification

Plumbing (from the run log, the outer command vero assembles):

harbor run -p /…/vero-harbor-6zi3nb4v/task -a codex -e modal -m gpt-5.3-codex \
  --ek modal_vm_runtime=true --yes -o ../runs/tau3-vmruntime/jobs

An end-to-end tau3 run on Modal with this branch is in flight; it is a straight A/B against 2026-07-25__07-33-29 (identical config, only the flag differs). Result posted here when it lands.

Tests: test_harbor_run_forwards_build_declared_optimizer_args (outer command carries modal_vm_runtime=true, nested-only flags stay off it, build-declared flags precede CLI args) and an optimizer_harbor_args controlled-flag case in test_load_build_config_rejects_invalid_specs.

Refs #55.

Greptile Summary

This PR adds optimizer_harbor_args to HarborBuildConfig, allowing a build to inject flags into the outer harbor run that hosts the optimizer trial, as opposed to extra_harbor_args which targets only the nested evaluation sub-run. The tau3 build.yaml is updated with a detailed comment explaining why none of the attempted fixes resolved the underlying StreamTerminatedError and that the field remains unset there.

• New config field: optimizer_harbor_args: list[str] on _HarborEvaluationFields, with a validator that blocks the four flags vero harbor run controls (-p, -a, -e, -m).
• CLI forwarding: build-declared flags are inserted before CLI extra args in the assembled command, preserving last-value-wins semantics for --ek overrides.
• Tests: a new integration test verifies the outer command carries the build's flags, the nested-only flags stay out, and order is correct; existing HTTP-layer tests are updated to supply the new field on their SimpleNamespace stubs.

Confidence Score: 5/5

Safe to merge; the change is additive, the new field defaults to an empty list, existing builds are unaffected, and the ordering guarantee (build flags before CLI extra) is both code-tested and documented.

The field addition is backward-compatible (empty-list default), the validator correctly blocks the four flags the outer CLI command already injects, and the forwarding logic in cli.py is a single-line extend covered by a dedicated integration test. The tau3 build.yaml change is documentation-only.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/build/config.py	Adds optimizer_harbor_args field and validator; validator correctly blocks the four flags the outer CLI command owns (-p, -a, -e, -m), consistent with existing extra_harbor_args approach.
vero/src/vero/harbor/cli.py	Inserts config.optimizer_harbor_args before extra in the assembled command so CLI args take precedence; correctly placed after agent-env and gateway args.
vero/tests/test_v05_cli.py	New test test_harbor_run_forwards_build_declared_optimizer_args validates outer command carries optimizer flags, excludes nested-only flags, and checks ordering relative to CLI extra args.
vero/tests/test_v05_harbor_build.py	Adds controlled-flag rejection case for optimizer_harbor_args and a passing roundtrip assertion; clean.
vero/tests/test_v05_harbor_http.py	Minimal fixup: two existing SimpleNamespace stubs gain optimizer_harbor_args=[] to match the new field accessed in run_command.
harness-engineering-bench/tau3/baseline/build.yaml	Adds explanatory comment for the absent optimizer_harbor_args value, documenting all four failed experiments; no code-level change.

Sequence Diagram

sequenceDiagram
    participant User as User / CLI
    participant Vero as vero harbor run
    participant Config as build.yaml / HarborBuildConfig
    participant Harbor as harbor run (outer optimizer trial)
    participant InnerHarbor as harbor run (nested eval sub-run)

    User->>Vero: vero harbor run --config build.yaml --agent codex -m model [extra...]
    Vero->>Config: load_harbor_build_config()
    Config-->>Vero: optimizer_harbor_args, extra_harbor_args, agent_env, ...
    Vero->>Harbor: harbor run -p task -a codex -e modal -m model [optimizer_harbor_args...] [extra...]
    Harbor->>InnerHarbor: harbor run -p task -a agent -e env -m model [extra_harbor_args...] (per-case eval)

Loading

_{Reviews (5): Last reviewed commit: "test: teach the run-command config stubs..." | Re-trigger Greptile}

[shehabyasser-scale]

Live result: the fix did not work. modal_vm_runtime=true does not save the tau3 trial.

runs/tau3-vmruntime/jobs/2026-07-25__08-38-37, 08:38 → 09:18, $1.27:

"n_errored_trials": 1, "n_retries": 0,
"exception_stats": {"StreamTerminatedError": ["task__pnBLeta"]}

The flag itself plumbed through correctly, so the mechanism in this PR works:

harbor run -p …/task -a codex -e modal -m gpt-5.3-codex --ek modal_vm_runtime=true --yes …
Selected strategy: _ModalDinD
Using vm_runtime: True          <- and the host-networking warning is gone

But the trial still ends grpclib.exceptions.StreamTerminatedError: Connection lost, with a tail identical to the previous run's.

The one thing that changed is the most informative part

The failure moved phases:

	previous run (DinD)	this run (vm_runtime)
harbor frame	single_step.py:105 _run_verifier → verifier.py:199 verify	single_step.py:77 _run_agent → codex.py:1102 run
when	after codex finished, during the verifier	during the codex run itself
bottom of stack	modal.py:1304 _sdk_exec → StreamTerminatedError	identical

Same bottom of stack, different long-running exec. That rules out my hypothesis. This is not about host networking or which exec happens to run at teardown — it is that harbor runs each phase as one exec and streams its stdout for the phase's entire duration, and a ~35-minute stream does not survive. Modal's own _stream_stdio_with_retries (task_command_router_client.py:838) tries to recover and re-raises.

So the original version of this PR had the right instinct and the wrong layer: retry is the applicable lever, it just has to reach the outer trial. The outer harbor run receives no --max-retries, which is why both runs log retries=0 and "Not retrying trial because the maximum number of retries has been reached".

What I propose keeping

• optimizer_harbor_args stays. It is verified working, it is the only way to configure the outer trial at all, and it is now the mechanism the actual fix needs.
• tau3's --ek modal_vm_runtime=true should come out, or stay only on its own merits (it does remove the host-networking caveat), not as a fix for this. I am not going to leave a value in a committed config claiming to fix something a live run says it does not fix.

Next experiment, not yet run

optimizer_harbor_args: ["--max-retries", "1"]. Honest caveat before spending it: a retry re-runs the whole ~40-minute codex trial, so worst case is ~80 minutes and ~$2.5, and it only pays off if the stream drop is probabilistic. Two for two so far, at different phases, is weakly consistent with "random drop over a long window" — but two samples is two samples. Flagging the cost rather than burning it unasked.

[shehabyasser-scale]

Final: all four experiments failed. This PR is now plumbing only.

#	outer trial config	elapsed	outcome
1	default (DinD)	39m55s	StreamTerminatedError in the verifier exec
2	--ek modal_vm_runtime=true	39m45s	StreamTerminatedError in the codex agent exec
3	--ek modal_sandbox_v2=true	38m14s	StreamTerminatedError
4	--max-retries 1	58m38s	StreamTerminatedError on both attempts

~$5.30 and four runs. tau3's optimizer trial now has no optimizer_harbor_args value, because none of these is a fix and a committed value implying otherwise is worse than an empty one.

What the experiments did establish

The mechanism works. Run 4 logged:

Trial task__XQNGEdr failed with exception StreamTerminatedError. Retrying in 1.00 seconds...

Every earlier run logged Not retrying trial because the maximum number of retries has been reached. So optimizer_harbor_args genuinely reaches the outer harbor run, and the outer trial is configurable for the first time. That is what this PR is for, and it stands on its own.

Retry is not the cure. The second attempt died identically. On this workload retry converts one guaranteed loss into two, at double the cost.

It is not a networking mode. vm_runtime moved the failure between phases while keeping an identical bottom of stack; sandbox_v2 changed nothing.

It looks deterministic, not flaky. 39m55s, 39m45s and 38m14s across three independent runs is far too tight a spread for a random gRPC drop. That is the single most useful thing these runs produced, and it argues against every "add resilience" fix including the one I proposed.

Where the real cause probably lives

Harbor runs each trial phase as one exec and streams its stdout for the phase's entire duration (codex.py:1102 → base.py:592 exec_as_agent → modal.py:1304 _sdk_exec), so any phase longer than ~40 minutes has a single point of failure with no partial credit. I looked for a matching deadline and did not find one: harbor's agent/verifier defaults are 600s, sandbox_timeout_secs defaults to 24h, and vero's compiled task.toml sets no [agent] timeout_sec at all. So whatever ends it at ~40 minutes is not in harbor's config surface, which is why no --ek fixed it.

This should go upstream to harbor as "long-running agent phases die at ~40 min on the Modal DinD/VM exec stream", with these four tracebacks attached. I have stopped spending on it here.

Review note

The one remaining judgement call is whether optimizer_harbor_args earns its place with no benchmark currently setting it. I think yes: without it there is no way to pass anything to the outer trial, and diagnosing this at all required exactly that. But it is a fair thing to push back on, and I would rather you decide that than have me quietly justify the field with a value that does not work.

[shehabyasser-scale]

Restacked. optimizer_harbor_args landed adjacent to agent_env, which f4e2057 cherry-picked from my fix/forward-agent-env-to-optimizer, so both the field block and the run_command call site conflicted. Resolved by keeping both; they are orthogonal (--ae populates the agent's env, this appends outer --ek flags) and the ordering only matters for flag precedence, which neither contests. Added the field to the Attributes: docstring per the convention 145fa29 introduced.

Also needed a test fix that is worth knowing about generally: test_v05_harbor_http.py builds the build-config as a SimpleNamespace carrying only the attributes run_command reads, so any new field breaks two tests with an AttributeError that surfaces only as a non-zero exit code with empty output. Same thing happened when agent_env was added. Stub updated in both directions.

The PR remains plumbing-only and no benchmark sets the field. The four live experiments behind the original tau3 justification all failed identically (default DinD 39m55s, modal_vm_runtime 39m45s, modal_sandbox_v2 38m14s, --max-retries 1 58m38s on both attempts), every one ending in grpclib StreamTerminatedError. The ~38-40 min consistency across three independent runs is the load-bearing datum: that is deterministic, not a flaky stream drop, so resilience flags are the wrong family of fix. Nothing in pr3-harness-bench touches it either, so it is still open and still belongs upstream in harbor.

That leaves the honest question unchanged: whether a config field earns its place with no caller. I would rather you decide that than have me justify it with a value that does not work.

Full suite matches the base branch: 11 pre-existing failures, none new.