3/3 Add harness-engineering-bench (GAIA + benchmarks) by varunursekar · Pull Request #46 · scaleapi/vero

varunursekar · 2026-07-23T17:57:21Z

Third of three stacked PRs splitting #41. Base: pr2-add-vero.

Adds harness-engineering-bench/ with each benchmark as a top-level sibling (gaia, ale-bench, officeqa, swe-atlas-qna, tau3) + scripts. Pure adds (71 files).

Union of 1/3 + 2/3 + 3/3 == v0.5 plus a legacy/ archive (verified).

🤖 Generated with Claude Code

Greptile Summary

This PR (3/3 of a stacked set) adds six benchmark harnesses under harness-engineering-bench/ — GAIA, OfficeQA, SWE-Atlas-QnA, tau3, BrowseComp-Plus, and SWE-bench-Pro — plus supporting scripts and a batch of vero framework improvements.

SWE-bench-Pro is a fully new benchmark and its baseline agent is the most robust in the suite: it ships with retry-with-backoff on transient API errors, parallel_tool_calls: False, correct context.metadata on every exit path, and a graceful turn-budget-exhausted path that doesn't raise or discard edits.
vero framework additions include a preflight model-availability check (_preflight_models), a new UPSTREAM_MODEL_UNAVAILABLE error taxonomy category, optimizer_harbor_args for the outer trial environment, a normalize_wandb_base_url fix that prevents scheme-less W&B hosts from silently disabling all reporting, and an artifact recording the W&B init failure when it occurs.
Several issues flagged in earlier reviews of this PR (GAIA turn-budget crash, GAIA _read_image propagating uncaught, per_trial_tokens.py None key) are resolved in the HEAD commit.

Confidence Score: 3/5

The vero framework additions and the SWE-bench-Pro harness are solid, but several benchmark agents introduced in this PR still carry open issues flagged in earlier reviews that have not yet been addressed.

The new SWE-bench-Pro agent, all vero source changes, and the admin scripts are clean and well-tested. Three previously flagged issues were fixed in the latest commit (GAIA turn-budget crash, GAIA _read_image propagation, per_trial_tokens None key). However, the OfficeQA, SWE-Atlas-QnA, BrowseComp-Plus, and tau3 baseline agents still carry a cluster of open defects — empty-output RuntimeErrors that abort trials, missing KeyError guards on Chat Completions tool arguments without strict mode, the tau3 MCP session-ID shell injection, duplicate YAML key in officeqa build.yaml, and the BrowseComp-Plus verifier format-string crash. These affect benchmark measurement correctness and agent robustness across four of the six benchmarks.

Files Needing Attention: harness-engineering-bench/tau3/baseline/target/src/tau3_agent/agent.py, harness-engineering-bench/officeqa/baseline/target/src/officeqa_agent/agent.py, harness-engineering-bench/swe-atlas-qna/baseline/target/src/atlas_agent/agent.py, harness-engineering-bench/browsecomp-plus/baseline/target/src/browsecomp_plus_agent/agent.py, harness-engineering-bench/browsecomp-plus/task-template/tests/evaluate.py, harness-engineering-bench/officeqa/baseline/build.yaml

Important Files Changed

Filename	Overview
harness-engineering-bench/swe-bench-pro/baseline/target/src/swebench_pro_agent/agent.py	New SWE-bench-Pro baseline agent; the most robust in the suite — includes retry-with-backoff, parallel_tool_calls disabled, correct context.metadata on all exit paths, graceful turn-budget else-branch, and path-traversal sanitization for write_file.
vero/src/vero/harbor/cli.py	Adds _preflight_models to probe configured models before launching a run; logic correctly distinguishes route-level 404s from model-level 404s, falls back to bare model name before blocking, and never blocks on inconclusive upstream failures. Also threads optimizer_harbor_args into the outer harbor run command.
vero/src/vero/evaluation/scoring/error_taxonomy.py	Adds UPSTREAM_MODEL_UNAVAILABLE category and two new TRANSIENT_INFRA patterns (sandbox/container loss, missing deployment); priority ordering in classify_case puts the deterministic operator-fixable error above co-occurring infra noise.
vero/src/vero/runtime/wandb.py	Adds normalize_wandb_base_url to prepend https:// to scheme-less WANDB_BASE_URL values before W&B parses them; called in both _open_wandb_run and WandbEventSink.init.
vero/src/vero/harbor/deployment.py	When the W&B sidecar sink fails to initialize, now writes the failure reason to session artifacts so a silently disabled sink is distinguishable from a healthy run that logged nothing.
vero/src/vero/harbor/build/config.py	Adds optimizer_harbor_args field with a validator that blocks controlled flags (-a/-e/-m/-p); symmetric with the existing extra_harbor_args validator.
harness-engineering-bench/scripts/per_trial_tokens.py	Previously flagged None-key crash on sorted() is fixed: evaluation_id now falls back to 'unattributed' rather than None, making all dict keys strings.
harness-engineering-bench/gaia/baseline/target/src/gaia_agent/agent.py	Fixes both previously flagged issues: turn-budget exhaustion now has a forced-final else branch, and tool dispatch is wrapped in a broad try/except (including OSError) so _read_image download failures feed an error to the model instead of crashing the trial.
harness-engineering-bench/scripts/rescore_candidate.py	New admin script to re-score a candidate from a failed verifier run; mirrors the baseline aggregation method so results are directly comparable to pinned baselines.
vero/tests/test_v05_cli.py	New tests covering preflight probe logic (route vs model 404 discrimination, fallback to bare name, inconclusive failures), optimizer_harbor_args forwarding, and opencode gateway argument construction.
vero/tests/test_v05_benchmark_configs.py	New integration tests asserting invariants across all six benchmark build.yaml files: gateway routing, model allow-lists, budget caps, and independent producer/evaluation scopes.

Sequence Diagram

sequenceDiagram
    participant CLI as vero harbor run
    participant PF as _preflight_models
    participant UP as Upstream API
    participant Harbor as harbor run
    participant Agent as Benchmark Agent
    participant GW as Inference Gateway
    participant Verifier as Task Verifier

    CLI->>PF: probe configured models (one token each)
    loop per model name
        PF->>UP: POST /responses (or /chat/completions)
        alt model-level 404 (DeploymentNotFound)
            UP-->>PF: 404 + error body
            PF-->>CLI: raise ClickException (block run)
        else route-level 404 or inconclusive
            UP-->>PF: 404 (no model mention) / timeout
            PF->>PF: try bare name fallback
        else 200 / non-404 error
            UP-->>PF: success or 4xx/5xx
            PF-->>CLI: continue (not fatal)
        end
    end
    CLI->>Harbor: launch with optimizer_harbor_args + extra
    Harbor->>Agent: run(instruction, environment)
    Agent->>GW: inference requests (metered, allow-listed)
    GW->>UP: forwarded requests
    UP-->>GW: responses
    GW-->>Agent: responses
    Agent->>Verifier: edits repo / writes answer.txt
    Verifier-->>Harbor: reward score
    Harbor-->>CLI: trial result

_{Reviews (60): Last reviewed commit: "Let the rescore script measure a benchma..." | Re-trigger Greptile}

socket-security · 2026-07-23T17:57:54Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/tqdm@4.68.4
	pypi/tqdm@4.69.0
	pypi/tqdm@4.69.1
	pypi/click@8.4.2
	pypi/uvicorn@0.51.0
	pypi/tiktoken@0.13.0
	pypi/requests@2.34.2
	pypi/fastapi@0.139.2
	pypi/fastapi@0.140.0

View full report

shehabyasser-scale · 2026-07-24T11:36:12Z

Approve with nits. The split is the part I care most about for experiment validity, and it checks out.

GAIA split is genuinely leak-safe (verified):

Deterministic stratified partition keyed by sha256("vero-gaia-v1:{task}") (partition_gaia.py:116), no RNG, with a --check mode that fails CI if the committed manifest drifts.
Verified disjoint: development 33 / validation 66 / test 66, zero pairwise overlap, union 165.
Wiring is correct: development at disclosure: full, validation at aggregate (min_aggregate_cases: 5, expose_case_resources: false), selection on validation, and reward/target on the test partition which is never agent-visible. A proper held-out test set.
k-anonymity floors are safe in every build.yaml (tau3 / officeqa / swe-atlas-qna / gaia all set 5). The ale-bench endswith("ACCEPTED") judge is fine too: upstream JudgeResult has no NOT_ACCEPTED member, so no failing verdict ends in that string.

One real defect (baseline quality, not integrity): the tau3 baseline drops conversation history after every plain-text turn. tau3_agent/agent.py:370 sets previous_response_id = None in the non-tool branch, while the tool branch threads it correctly (:347). After any non-terminal customer reply the model loses the domain policy (only present in the turn-0 input), prior tool results, and reasoning state, which depresses tau3 from the first follow-up onward. The Responses API supports threading after a message output, so None isn't required. Suggest previous_response_id = response.id on the text branch as well. Confirmed tau3-specific; the GAIA baseline has no such reset.

One thing to note for readers: every build.yaml ships infrastructure_max_attempts: 3 + aggregate_attempts: best, so the competitive-selection integrity of these benchmarks depends on the aggregate/retry fix over on #45, not on any change here.

… Claude The recommended launch was backwards. It advised `openai/claude-sonnet-5`, which crashes: that form forces the Responses API, and litellm's Anthropic->Responses translation emits three id namespaces in one stream (a resp_ id, Anthropic msg_/toolu_ item ids, and a stray chatcmpl- id), so opencode dies resolving a text part under an id it never registered. All six main-model calls returned 200, so the gateway was never at fault. `anthropic/claude-sonnet-5` is now the right form. It previously escaped the gateway -- harbor's adapter injects a baseURL only for the openai provider and opencode ignores ANTHROPIC_BASE_URL -- but vero now supplies that baseURL itself for any non-openai provider. Measured on the conformance example: reward 1.0 over 39 steps, 39 metered `messages` calls, no translation layer involved. Also corrects a security claim that overstated the risk. Escaping the gateway fails closed rather than leaking: the optimizer only ever holds a scoped producer token, so a direct call to a provider's public endpoint returns `401 invalid x-api-key` and the run dies. The upstream credential never leaves the gateway container. The note previously said such a run would hold a credential the optimizer is never supposed to see, which would have discouraged the form that now works, for the wrong reason. Adds two things the same investigation turned up: opencode issues an auxiliary small-model call of the same provider family (claude-haiku-4-5 or gpt-5.4-nano) that 403s against a single-entry allow-list, silently and invisibly outside the gateway log; and it drives the stateful Responses API for openai/ providers, which changes how per_trial_tokens.py's content-matching fallback behaves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…t user `uv tool install` symlinks a harness's entry point into /usr/local/bin, which the optimizer user cannot write, so the install step dies with "Failed to install executable ... Permission denied (os error 13)" and the trial ends in NonZeroAgentExitCodeError before any evaluation runs. Pointing UV_TOOL_BIN_DIR at the user's own bin directory fixes it. This blocks both literature-standard harnesses -- mini-swe-agent and swe-agent are each installed this way -- while claude-code and opencode are npm/nvm-based and never hit it, so the failure only appears when the harness dimension of the grid changes, and reads like "this harness does not work with vero" rather than a container-permissions problem. Found by vero/examples/harness-conformance in 91 seconds; mini-swe-agent then scored 1.0 on the same check in 4m57s, with 25 producer requests all 200 on chat/completions and no denials. Applied to all five benchmarks rather than the one about to be run, since the first mini-swe-agent cell of the grid would otherwise hit it whichever benchmark came first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

gaia and officeqa already log to harness-engineering-bench with a per-benchmark group and a ${wandb_run} launch label; swe-atlas-qna, tau3 and browsecomp-plus were still on their own vero-<benchmark> projects, which would have scattered the grid across five places. Verified by compile. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Learned by losing one. officeqa run #4 was killed while the verifier was blocked on a drain, so it never reached export-session -- and stopping the compose stack removes the containers, taking /state/admin/session with them. Every commit that optimizer made, including the candidate it had deliberately submitted after a 5.5h search, was unrecoverable. The verifier/ directory it left behind holds a single empty file. The fix is one docker cp before the kill, which yields candidates/repository.git and lets scripts/rescore_candidate.py score the submitted candidate on the held-out set afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The previous note claimed the binding limit was a shared Fireworks account ceiling of 216k generated tokens/min, capping the suite near 2 concurrent runs no matter how many keys were used. That was wrong, and it was wrong in the direction that would have throttled the whole week's schedule. It came from misreading the llm_provider-x-ratelimit-* headers as a depleting account bucket. They are the provider's limits echoed per request. The measurement that disproves it: remaining-tokens-prompt reads 14062495 both on an idle probe and while ~120 cases were in flight -- byte-identical, and in both cases exactly limit minus 5, where 5 is that probe's own prompt tokens. Over the same interval the x-ratelimit-api_key-* counters moved as a real bucket should, 4999 -> 4863 requests and 10M -> 9.52M tokens. So the binding ceiling is the per-key bucket, 10M TPM and 5000 RPM. One run peaks at 2.90M TPM and ~182 RPM, which is roughly three concurrent runs per key, and adding keys scales parallelism proportionally -- two keys cover the suite, three leave headroom. Target models still differ per benchmark, which spreads load across providers, but that is a side benefit rather than the constraint. Fix the same claim in secrets.env.example. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Concurrency planning is bounded by the LiteLLM per-key bucket, and we have twice now reasoned about that bucket from assumption rather than measurement -- once concluding the ceiling was a shared provider limit that keys could not raise, which was wrong. This makes the check one command: python3 harness-engineering-bench/scripts/check_keys.py vero/ For every *.secrets.env it sends one 1-token request and reports the key's actual TPM/RPM alongside how many concurrent runs that carries, using the measured per-run draw (2.90M TPM / ~182 RPM for officeqa at max_concurrency 24). Keys are shown as an 8-char fingerprint, never in full. It also states the two ceilings the key limits do not cover -- local capacity (~8-10 runs on a 14-core box, since the gateway proxies every evaluation token through a local process) and Modal sandbox concurrency (runs x max_concurrency) -- so raising a key limit is not mistaken for raising everything. Verified against the current credential: 10M TPM / 5000 RPM -> 3 runs, matching the hand measurement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A key still reporting its old ceiling right after a change is probably stale rather than unchanged, and reading it as authoritative would under-plan concurrency. Say so in the checker's output and docstring. Also correct the local-capacity figure printed alongside. The earlier ~8-10 runs was extrapolated from a 2-run sample; measured, 4 concurrent runs sat at load 3.11 on 14 cores and load did not rise at all going from 3 to 5 workloads, because the work happens in Modal sandboxes and the local gateway is I/O-bound. ~15-20 runs is the honest number. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

WIP scaffold: a working agent + build/config skeleton for a SWE-bench-Pro optimization target, mirroring the gaia baseline layout. Two things are stubbed and must be completed before it runs: (1) the task_source digest in baseline/build.yaml is a clearly-marked placeholder that must be pinned to the real SWE-bench-Pro Harbor task package, and (2) the partitions/ files are empty placeholders that must be regenerated from the real task list via scripts/partition_swe_bench_pro.py. The agent wraps responses.create in a retry-with-backoff helper from the start (the unguarded call is what scored the gaia baseline 0.0). See harness-engineering-bench/swe-bench-pro/README.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The agents strip only the `openai/` prefix before calling the gateway (`removeprefix("openai/")`), and the gateway matches the allow-list exactly (`model not in scope.allowed_models` -> 403 model_denied). So `model: openai/gpt-4o` puts `gpt-4o` on the wire, and an allow-list carrying the prefixed spelling never matches. Listing both spellings papered over that and then fails the config invariant that the evaluation scope allow exactly the target model. Drop the prefix on both sides, and template the producer list the way every other benchmark does so `--model` / `--param optimizer_model` keeps working without a config edit. This config still cannot be swept by tests/test_v05_benchmark_configs.py: its partitions are deliberately empty pending real task ids, so it fails to load. It should join BENCHMARKS there once the partitions and the task source digest are pinned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A model that is configured but not deployed upstream is invisible until the run is over. The gateway forwards the request, the upstream 404s, the agent makes no progress, and every case is scored 0.0 and labelled `task_failure` — a run that looks honest and says the candidate failed. That is how swe-atlas-qna produced 469 zero-score cases across two runs (#51): `gpt-5.4-mini-2026-03-17` is not provisioned on the configured Azure resource, so 322/322 evaluation-scope and 79/79 finalization-scope requests returned 404 DeploymentNotFound while the producer scope ran clean. `vero harbor run` now sends one 16-token request per distinct allow-listed model before compiling the task, and aborts if any comes back 404. Only a definitive 404 is fatal. A timeout, a 429 or a 503 is inconclusive and is reported without blocking, so a transient upstream blip cannot stop a run that would have succeeded. Names are probed with the provider prefix stripped, which is the form the agents actually send. Verified against the live endpoint: gaia (which points at the undeployed model) now exits 1 in about two seconds with the DeploymentNotFound body quoted, instead of burning ~30 minutes and ~$2; a gpt-5.3-codex producer with a gpt-4o evaluation scope passes. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The probe only spoke the Responses API and treated any 404 as proof the deployment was absent. A route-level 404 (the upstream does not implement that path) and a model-level 404 (the deployment does not exist) share a status code and mean opposite things, so the preflight would hard-block a run against a Chat-Completions-only upstream that would have succeeded. That is now a live risk in this repo: officeqa, tau3, swe-atlas-qna and browsecomp-plus were ported to Chat Completions, leaving gaia as the only agent still on Responses. Try both routes, and only call a model missing when the 404 body names the model (`DeploymentNotFound`, `model_not_found`, or "model ... does not exist") rather than the route. If every route 404s on the route itself, the result is inconclusive and the run proceeds. Same fix for the model name: the provider prefix routes on a proxy and is meaningless to a single-provider endpoint, so probe the configured spelling first and fall back to the bare one before reporting anything missing. Previously the prefix was stripped unconditionally, which is wrong for `fireworks_ai/...` names. Verified against the live Azure endpoint: `gpt-4o` 200 on both routes; `openai/gpt-5.3-codex` 404s prefixed and passes preflight via the bare fallback; `fireworks_ai/gpt-oss-120b` is still correctly blocked; and a model 404 body is distinguished from a bare route 404. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The harbor backend's error taxonomy records any unrecognized case failure as TASK_FAILURE: an informative, scoreable sample at the failure value with status SUCCESS. That is right for a candidate that crashes on its own, but Modal sandbox lifecycle failures and a missing held-out tests fixture are infrastructure the candidate did not cause. Seen on a swe-atlas-qna run (2026-07-24): all 396 cases returned status=success, error_category=task_failure, score=0.0. Terminal exceptions were AddTestsDirError x144, Modal sandbox NotFound x125, RuntimeError x67, BadRequestError x60. Classified as task failures, the aggregate reported error_rate 0.0 and objective feasible at 0.0, so a fully broken environment looked like a real, shippable zero and the optimizer ran with no gradient (baseline and every candidate 0.0). Add the harness/Modal environment-loss signals (sandbox lifecycle, AddTestsDirError / "failed to add tests directory", container fetch failures, StreamTerminatedError) to the TRANSIENT_INFRA patterns. Those cases are now excluded from the aggregate and counted toward invalidity, so a collapsed environment surfaces as an invalid run instead of a silent zero. A candidate's own bug (ValueError, KeyError, no answer) is still a task failure, unchanged. Covered by two new tests; existing behavior asserted unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

#53 stopped a dead sandbox being scored as an informative zero. The same masking survives one layer up, for a cause that is even easier to fix. In the 2026-07-25 swe-atlas-qna run, 145 of 469 cases ended on: NotFoundError: Error code: 404 - {'error': {'type': 'invalid_request_error', 'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. ...'}} The configured eval model is not provisioned on the Azure resource. The agent's every call 404s, it writes no answer, and `classify_case` — finding nothing in the signal it recognises — returns TASK_FAILURE, whose policy is `is_informative_sample=True`. So each case was recorded as a genuine score of 0.0 and the harness reported that the candidate failed the task. Add UPSTREAM_MODEL_UNAVAILABLE: never an informative sample, never retried, terminating (every remaining case fails identically, so there is nothing left to measure), and counted toward invalidity so the aggregate still comes out invalid if the terminating path is ever bypassed. It is ranked above TRANSIENT_INFRA, so a case that saw both a dead sandbox and a missing deployment reports the deterministic, operator-fixable cause. The pattern deliberately matches the provider error codes and the exact Azure sentence rather than a bare "does not exist": 102 cases in that same run failed with `FetchSpec failed: loading container: file does not exist`, which is genuine infrastructure and must stay TRANSIENT_INFRA. There is a test pinning both sides. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The scaffold could not resolve a task: `task_source` was the literal `scale-ai/swe-bench-pro@sha256:REPLACE_WITH_REAL_DIGEST`, the three partition files were empty, and the split script refused to run behind a `TOTAL_TASKS = 0` stub guard. SWE-bench-Pro is not a package dataset. It ships as `swebenchpro` in Harbor's default registry (731 instances over 11 projects), whose tasks resolve to git-backed ids under laude-institute/harbor-datasets at commit c8e8f3fac7097accaacf261d74c3d6f441de45b1, so the pin is a bare `name@version` rather than a content digest. That distinction drives the rest of the change: - `_read_tasks` reads the stratification key from `tests/config.json` (`repo`), not `[metadata].repository`: these task.toml files carry no `[task].name`, and Harbor derives the canonical name from the task directory, which is what `-i` matches. - `_fetch_registry_refs` goes through `RegistryClientFactory` instead of `PackageDatasetClient`, recording `<git_commit>:<path>` as the per-task ref, which identifies content the way a digest would. - `expose_case_resources` is false on the development partition. VeRO materializes case resources through `PackageDatasetClient`, which only accepts `<org>/<name>` refs, so a registry dataset cannot be exposed that way. - 731 splits 20/40/40 as 146.2/292.4/292.4; the leftover goes by largest remainder and the .4/.4 tie breaks toward test, matching how officeqa (246 -> 49/98/99) and swe-atlas-qna (124 -> 25/49/50) resolved it. Hence 146/292/293. - `task_agent_timeout_seconds` drops 3600 -> 3000 to mirror the task package's own `[agent] timeout_sec`, making the runner's --agent-timeout-multiplier exactly 1800/3000 = 0.6. - `requires-python` floors at 3.12 because harbor 0.20.0 requires it; ">=3.11" left the package's own `uv run pytest` unresolvable. The <3.14 cap stays (litellm/pyo3 fails to build on 3.14). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ward Each of these silently biases the measurement rather than failing loudly, so the optimizer would have hill-climbed against a broken baseline. 1. REPO_DIR was `/app/repo`. The swebenchpro task images set `WORKDIR /app` and reset the checkout in place there, and the task instruction says so verbatim ("I've uploaded a code repository in the directory /app"). Every shell command ran in a directory that does not exist. 2. `reasoning: {effort: high}` was sent unconditionally. Non-reasoning models reject it with a hard 400 on the very first turn, so a gpt-4o run could not complete a single trial. Gated on capability, matching the sibling agents. 3. `_write_file` joined a model-supplied path onto `logs_dir`. Models routinely answer with the absolute path they just saw in shell output, and `Path(logs_dir) / "staged" / "/app/x.py"` discards the left side entirely, so the write landed on the HOST root and killed the trial with `OSError [Errno 30] Read-only file system: '/app'`. Seen on 2 of the first 12 held-out trials. Paths are now normalized repo-relative and traversal is refused. 4. Exhausting the turn budget raised RuntimeError. Reward comes from the task's hidden suite run against whatever the agent left in the repository, so raising discarded real edits, errored the trial, made harbor re-run the whole thing (9 of the first 49 held-out trials) and scored a hard 0 for a fix that may well have been correct. It now records the exhaustion in metadata and lets the verifier grade, which is exactly what the "model stopped calling tools" branch already did. 5. Harbor decodes exec output as strict UTF-8. These are real repositories, so a model that cats or greps a binary raised UnicodeDecodeError out of `exec` and killed the trial: 5 of the first 49 held-out trials. A non-decodable stream is now a tool error the model can react to, with a hint to re-run through `strings`/`file`. Verified end to end: a single-case smoke on instance_ansible__ansible-0ea40e with a gpt-4o agent completed with reward 1.0, all 16 required tests PASSED, 0 errored trials. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The config-invariant suite enumerates benchmarks explicitly, so swe-bench-pro was silently unchecked. Now that its task_source resolves, it can join the other five. All three parametrized cases pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

gaia and swe-bench-pro send `reasoning: {effort: ...}` unconditionally. A non-reasoning model rejects it with HTTP 400, so every call fails and the case is scored as an honest-looking task failure. Measured on a live gaia run: 715 of 2376 calls returned 400 (514 evaluation, 201 finalization). Gate it on the model being reasoning-capable. No behavior change for reasoning models; gpt-4o stops 400ing. Scoped to the two agents that have no gate at all. officeqa, tau3 and swe-atlas-qna gained `if "fireworks" not in self._api_model` in b7a5b80, so their hunks are dropped here rather than rewritten. That gate is provider-shaped rather than capability-shaped and still sends reasoning_effort to any non-Fireworks non-reasoning model such as Azure gpt-4o, which is raised as review on the PR rather than changed unilaterally. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Restores the hunks dropped when this PR was restacked. b7a5b80 added `if "fireworks" not in self._api_model` to officeqa, tau3 and swe-atlas-qna, which is provider-shaped rather than capability-shaped: it correctly spares Fireworks-served open models and still sends `reasoning_effort` to any non-Fireworks model that cannot accept it. That is not theoretical. Measured today against the configured Azure endpoint, running the swe-atlas-qna seed agent over its 50-case held-out split: gpt-5.3-codex, provider gate 50/50 BadRequestError 400 "The requested operation is unsupported" (that model has no chat/completions surface on this resource at all) gpt-4o, provider gate 50/50 BadRequestError 400 "Unrecognized request argument supplied: reasoning_effort" gpt-4o, capability gate 0 BadRequestError, agent inference succeeded So the provider gate makes these three benchmarks unrunnable against any non-Fireworks upstream, which is every endpoint we currently have. The capability form also excludes `fireworks_ai/*`, so it is a strict superset of the current behaviour and changes nothing for the default configuration. officeqa gates two arguments on that one condition. Only `reasoning_effort` moves: `parallel_tool_calls` is a separate axis that Fireworks rejects and gpt-4o supports, so it keeps the provider check. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The four Chat Completions agents send `max_tokens` unconditionally. Every gpt-5 model rejects it: "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead." So even with the reasoning_effort gate corrected, these agents could not target any modern OpenAI reasoning model, only Fireworks-served models and gpt-4o. Select the parameter name from the same capability test, and fold the two copies of that test into one `_is_reasoning_model` helper per agent so they cannot drift apart. browsecomp-plus's reasoning_effort check moves onto the helper as well; it had the same provider-shaped gate as the other three. Verified live against the configured Azure endpoint, sending exactly the shapes the patched agents now build: gpt-4o ['max_tokens'] 200 gpt-5.4-mini-2026-03-17 ['max_completion_tokens','reasoning_effort'] 200 and the predicate classifies the suite's own targets correctly: `fireworks_ai/deepseek-v4-flash` and `fireworks_ai/gpt-oss-120b` are both non-reasoning, so the default configuration keeps the legacy shape and is unaffected. gaia and swe-bench-pro are untouched here: they use the Responses API, whose `max_output_tokens` has no such split. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The tau3 optimizer trial kept dying ~22 min in, after codex had finished, with `grpclib StreamTerminatedError: Connection lost` raised from harbor's finalization verifier exec (harbor/verifier/verifier.py:199 -> environments/dind_compose.py -> modal.py `_sdk_exec`). Harbor logs its own remedy at trial start: "DinD mode uses host networking ... Use --ek modal_vm_runtime=true to use the VM runtime instead". There was no way to declare that. `extra_harbor_args` reaches the NESTED `harbor run` that scores a candidate (build/config.py -> HarborBackendConfig -> backend.py `_harbor_command`), not the OUTER `harbor run` that hosts the optimizer trial, which is where the stream drops. Add `optimizer_harbor_args`, forwarded by `vero harbor run` to the outer command ahead of any trailing CLI args (harbor's `--ek` is last-value-wins, so the command line still overrides), and set tau3's to `--ek modal_vm_runtime=true`. This supersedes the previous version of this PR, which raised tau3's `max_retries` 1 -> 3. That knob is the nested per-case retry and was proven not to be the failing layer: the 2026-07-25 run still errored with `retries=0` and "Not retrying trial because the maximum number of retries has been reached", because the outer trial never receives `--max-retries`. Reverted to 1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Four live runs, each ~40 min on Modal, all ending the same way: default DinD 08:13 StreamTerminatedError (verifier exec) modal_vm_runtime 09:18 StreamTerminatedError (codex agent exec) modal_sandbox_v2 10:02 StreamTerminatedError --max-retries 1 10:22 StreamTerminatedError on both attempts So tau3's optimizer trial gets no `optimizer_harbor_args` value. None of the four is a fix, and a committed config value that implies otherwise is worse than an empty one. What the experiments did establish: - The mechanism works. With `--max-retries 1` the trial logged "failed with exception StreamTerminatedError. Retrying in 1.00 seconds" where every earlier run logged "Not retrying ... maximum number of retries has been reached". The outer layer is now configurable, which it was not before. - Retry is not a cure here. The second attempt died the same way, so this converts one guaranteed loss into two. - It is not a networking mode. The failure moved between phases under vm_runtime but kept an identical bottom of stack, and sandbox_v2 changed nothing. - It looks deterministic, not flaky: 39m55s, 39m45s and 38m14s to failure across three independent runs is too tight for a random stream drop. The common factor is that harbor runs each phase as one exec and streams its stdout for that phase's whole duration. No harbor timeout matches ~40 min, so the deadline, if there is one, is not in harbor's config surface. This PR is therefore just the `optimizer_harbor_args` plumbing plus its tests. The tau3 failure belongs upstream with harbor. Refs #55. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two tests in test_v05_harbor_http.py build the build-config as a SimpleNamespace carrying only the attributes run_command reads, so adding a field to HarborBuildConfig breaks them with an AttributeError that surfaces only as a non-zero exit code. Same convention as when agent_env was added: the stub grows with the config. The mirror of that also applied to this branch's own test, whose _Config predates agent_env and so broke once both fields were in the same code path. Suite now matches the base branch exactly: 11 pre-existing failures, none new. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Every harbor benchmark run has reported nothing to the self-hosted W&B: `shehab-yasser` listed zero projects, and each run's exported `artifacts/wandb/state.json` showed a run_id minted with `evaluation_ids: []`, `next_step: 0`, `request_log_files: {}` — even for a session that completed 11 evaluations. That fingerprint is exactly what `SidecarWandbSink.__init__` leaves when `wandb.init()` raises: state is written (wandb.py:225) before the run is opened (wandb.py:230), and `deployment.py` catches anything from the constructor and continues without W&B. Reproduced locally against scaleai.wandb.io, $0, deterministic: WANDB_BASE_URL=scaleai.wandb.io pydantic_core.ValidationError: 1 validation error for Settings base_url Input should be a valid URL, relative URL without a base [type=url_parsing, input_value='scaleai.wandb.io'] W&B parses `base_url` as a URL, so the natural way to write a self-hosted host silently costs the run all of its reporting. With `https://` prepended, the same script logs metrics, uploads an artifact and auto-creates the project. Egress was never the problem. Two changes: - `normalize_wandb_base_url()` prepends `https://` to a scheme-less `WANDB_BASE_URL` (and warns), called before both `wandb.init()` sites. Already-qualified values, including plain `http://localhost`, pass through. - The swallowed init failure now also lands in `session/artifacts/wandb/init-error.json`. The existing `logger.warning` goes to the sidecar container's stderr, which no run artifact captures, so a disabled sink was indistinguishable from a healthy run that logged nothing. Observability still never takes the eval path down. Refs #52. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

fix: only send reasoning.effort to models that support it

feat: let a build configure its outer optimizer trial's harbor flags

fix: repair a scheme-less WANDB_BASE_URL and stop losing W&B sink failures

…fold feat: make the SWE-bench-Pro baseline runnable for the vero optimizer

… candidate Re-pinning a baseline and re-scoring a candidate are the same measurement with a different starting tree, so --seed reuses the whole verified path rather than introducing a second scorer whose equivalence we would have to argue: plain harbor run over the explicit test task list, N rounds pooled, no gateway, and no --agent-timeout-multiplier so harbor's default 1.0 applies exactly as it did when the baselines were first pinned. Aggregation still matches runs/recompute.py. The seed is copied out of the benchmark's agent_repo before running so a measurement cannot mutate the checkout, and __pycache__/.venv/.git are excluded. Needed because the pinned baselines are of uncertain currency: gaia's seed scored 0.6414 through a run's finalization against a pinned 0.5736. Whether that is a stale pin or gaia's own variance is what re-measuring settles -- officeqa's first fresh round already lands at 0.3535 against its pinned 0.3603, which suggests the pins are healthier than the gaia figure implied. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

varunursekar mentioned this pull request Jul 23, 2026

Redesign VeRO as a generic program optimization harness #41

Closed

greptile-apps Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread harness-engineering-bench/ale-bench/environment/sidecar/dind-entrypoint.sh Outdated

Comment thread harness-engineering-bench/tau3/baseline/target/src/tau3_agent/agent.py

Comment thread harness-engineering-bench/officeqa/baseline/target/src/officeqa_agent/agent.py

varunursekar force-pushed the pr3-harness-bench branch 3 times, most recently from e0a1efb to ab4c52f Compare July 23, 2026 20:00

greptile-apps Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread harness-engineering-bench/tau3/baseline/target/src/tau3_agent/agent.py

varunursekar force-pushed the pr3-harness-bench branch 2 times, most recently from 573c1ac to c86351e Compare July 23, 2026 22:58

greptile-apps Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread harness-engineering-bench/ale-bench/environment/sidecar/harness/score.py Outdated

varunursekar force-pushed the pr3-harness-bench branch 3 times, most recently from 745ed8e to 6788d76 Compare July 24, 2026 03:49

greptile-apps Bot reviewed Jul 24, 2026

View reviewed changes

Comment thread harness-engineering-bench/officeqa/baseline/build.yaml Outdated

varunursekar force-pushed the pr3-harness-bench branch 4 times, most recently from 214f1aa to a2fcd19 Compare July 24, 2026 06:41

shehabyasser-scale approved these changes Jul 24, 2026

View reviewed changes

varunursekar force-pushed the pr3-harness-bench branch 4 times, most recently from 768de08 to ad2c081 Compare July 24, 2026 22:28

greptile-apps Bot reviewed Jul 25, 2026

View reviewed changes

Comment thread harness-engineering-bench/tau3/baseline/target/src/tau3_agent/agent.py

greptile-apps Bot reviewed Jul 25, 2026

View reviewed changes

Comment thread harness-engineering-bench/scripts/per_trial_tokens.py

greptile-apps Bot reviewed Jul 25, 2026

View reviewed changes

Comment thread harness-engineering-bench/officeqa/baseline/target/src/officeqa_agent/agent.py

varunursekar force-pushed the pr3-harness-bench branch from 741bfa6 to 95fed89 Compare July 25, 2026 19:52

greptile-apps Bot reviewed Jul 26, 2026

View reviewed changes

Comment thread harness-engineering-bench/browsecomp-plus/task-template/tests/evaluate.py

varunursekar force-pushed the pr3-harness-bench branch from 51c152d to ced19f9 Compare July 26, 2026 04:26

varunursekar and others added 4 commits July 27, 2026 18:19

varunursekar force-pushed the pr3-harness-bench branch from 31d8b79 to ab8717b Compare July 28, 2026 02:25

greptile-apps Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread harness-engineering-bench/officeqa/baseline/target/src/officeqa_agent/agent.py

varunursekar and others added 24 commits July 27, 2026 19:53

Merge pull request #54 from scaleapi/fix/judge-reasoning-effort

2aff9f4

fix: only send reasoning.effort to models that support it

Merge pull request #55 from scaleapi/fix/tau3-trial-resilience

c8b6871

feat: let a build configure its outer optimizer trial's harbor flags

Merge pull request #57 from scaleapi/fix/wandb-base-url-normalization

16fbc2a

fix: repair a scheme-less WANDB_BASE_URL and stop losing W&B sink failures

Merge pull request #50 from scaleapi/feat/swe-bench-pro-baseline-scaf…

d93c696

…fold feat: make the SWE-bench-Pro baseline runnable for the vero optimizer

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

3/3 Add harness-engineering-bench (GAIA + benchmarks) - #46

3/3 Add harness-engineering-bench (GAIA + benchmarks)#46
varunursekar wants to merge 81 commits into
pr2-add-verofrom
pr3-harness-bench

varunursekar commented Jul 23, 2026 •

edited by greptile-apps Bot

Loading

Uh oh!

socket-security Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shehabyasser-scale commented Jul 24, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

varunursekar commented Jul 23, 2026 • edited by greptile-apps Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 3/5

Important Files Changed

Sequence Diagram

Uh oh!

socket-security Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shehabyasser-scale commented Jul 24, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

varunursekar commented Jul 23, 2026 •

edited by greptile-apps Bot

Loading

socket-security Bot commented Jul 23, 2026 •

edited

Loading