fix: repair a scheme-less WANDB_BASE_URL and stop losing W&B sink failures

[shehabyasser-scale]

Closes #52 (pending the live confirmation noted at the bottom).

The symptom

No harbor benchmark run has ever reported anything to the self-hosted W&B. Entity shehab-yasser listed zero projects. Every run's exported session/artifacts/wandb/state.json looked like this:

{"evaluation_ids": [], "next_step": 0, "run_id": "vero-a25e1cd8b7064d7e", "request_log_files": {}}

…from the 2026-07-25 swe-atlas-qna run whose session contains 11 completed evaluations. So "nothing was logged" was never "nothing happened".

The mechanism

That exact fingerprint is what SidecarWandbSink.__init__ leaves behind when wandb.init() throws. _save_state() runs at runtime/wandb.py:225, _open_wandb_run() at :230. harbor/deployment.py:265-287 catches anything out of the constructor and continues with wandb_sink = None, by design — observability must not take the eval path down.

The cause, reproduced locally at $0

$ WANDB_BASE_URL=scaleai.wandb.io python -c "import wandb; wandb.init(project='vero-egress-probe', ...)"
pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
base_url
  Input should be a valid URL, relative URL without a base
    [type=url_parsing, input_value='scaleai.wandb.io', input_type=str]

W&B parses base_url as a URL. The natural way to write a self-hosted host is rejected, and the rejection is indistinguishable from "W&B is unavailable".

Same script with https:// prepended:

INIT OK -> https://scaleai.wandb.io/shehab-yasser/vero-egress-probe/runs/vero-probe-0001
wandb: Synced 5 W&B file(s), 0 media file(s), 2 artifact file(s)
FINISH OK

The project auto-created server-side; query_wandb_entity_projects(entity="shehab-yasser") went from [] to [vero-egress-probe]. Metrics logged, wandb.Artifact uploaded. Egress was never blocked, credentials were never missing, and the code was never unplumbed.

Note the repo's own swe-bench-pro/baseline/secrets.env.example:23 already gets it right (WANDB_BASE_URL=https://scaleai.wandb.io); the live gitignored secrets.env had drifted to the scheme-less form. Since the deployed secrets file is not in the repo, fixing only the file would leave the trap armed for the next operator.

Also worth correcting

Issue #52 is titled "Weave traces". There is no Weave in this codebase — weave is not a dependency and is not importable in the built env. wandb.log_traces drives SidecarWandbSink._log_trace, which uploads a wandb.Artifact of type evaluation_trace. So count_weave_traces would return nothing even on a perfectly healthy run; the right server-side check is whether the project exists and carries runs and artifacts.

Changes

• normalize_wandb_base_url() prepends https:// to a scheme-less WANDB_BASE_URL and warns; called before both wandb.init() sites. Values that already carry a scheme (including http://localhost:8080) pass through untouched; unset stays unset.
• The swallowed init failure now also lands in session/artifacts/wandb/init-error.json with the exception type and message. The existing logger.warning only reaches the sidecar container's stderr, which no run artifact captures — grepping every artifact of the failing run for "wandb" returns nothing. That invisibility is what made a one-line config bug survive multiple runs.

Tests: test_scheme_less_wandb_base_url_is_repaired_before_init (repair, passthrough, unset, and that a sink repairs before opening its run) and test_wandb_init_failure_is_recorded_in_the_session_artifacts (eval path survives, reason is durable).

Still to confirm

The local probe proves the URL fix and that scaleai.wandb.io is reachable from this laptop. Modal-sandbox egress to it is not yet proven. A live benchmark run on this branch is queued; I will post the resulting project URL and artifact count here before this merges.

Greptile Summary

This PR fixes two independent failure modes that caused every harbor benchmark run to report nothing to the self-hosted W&B instance: a scheme-less WANDB_BASE_URL that W&B's pydantic model rejected, and a swallowed init failure that left no durable evidence in run artifacts.

• normalize_wandb_base_url() prepends https:// to any scheme-less value and mutates os.environ in-place so W&B's settings model receives the corrected URL; it is called before both independent wandb.init() sites (_open_wandb_run for SidecarWandbSink, and directly in WandbEventSink.__init__).
• On construction failure, deployment.py now writes wandb/init-error.json into the session artifact directory (in addition to the existing stderr warning), making the failure discoverable via the exported run record rather than only through transient container logs.
• Two new tests cover scheme repair (including passthrough and unset), repair-before-run-open for SidecarWandbSink, eval-path survival on init failure, and durability of the recorded error.

Confidence Score: 5/5

Safe to merge once the live benchmark run confirms Modal-sandbox egress to scaleai.wandb.io (as noted in the PR description).

Both changes are narrow and defensive: URL normalization is idempotent and only fires on misconfigured input, and the error-recording path is wrapped in its own OSError guard so it can never affect the eval path. The existing sink-failure handling is preserved exactly; only the durability of the diagnosis is improved. Tests cover the new code paths directly.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename	Overview
vero/src/vero/runtime/wandb.py	Adds normalize_wandb_base_url() helper and calls it before both wandb.init() sites; the function is idempotent and correctly mutates os.environ so W&B's pydantic settings model picks up the corrected value.
vero/src/vero/harbor/deployment.py	On W&B sink construction failure, now writes wandb/init-error.json into the session artifacts in addition to the existing stderr warning; uses except OSError as a safe fallback so artifact-write errors cannot interrupt the eval path.
vero/tests/test_v05_wandb.py	Adds test_scheme_less_wandb_base_url_is_repaired_before_init covering scheme-less repair, already-qualified passthrough, unset, and repair-before-run-open for SidecarWandbSink.
vero/tests/test_v05_harbor_deployment.py	Adds test_wandb_init_failure_is_recorded_in_the_session_artifacts; monkey-patches SidecarWandbSink with an exploding callable, verifies the eval path survives and that wandb/init-error.json carries the project name, exception type, and message.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["WANDB_BASE_URL set in env"] --> B{normalize_wandb_base_url}
    B -->|"has '://' or empty"| C["pass through unchanged"]
    B -->|"scheme-less e.g. scaleai.wandb.io"| D["prepend https://\nlog warning\nmutate os.environ"]
    C --> E[wandb.init]
    D --> E

    subgraph SidecarWandbSink path
        F["SidecarWandbSink.__init__"] --> G["_save_state (run_id persisted)"]
        G --> H["_open_wandb_run()"]
        H --> B
    end

    subgraph WandbEventSink path
        I["WandbEventSink.__init__"] --> B
    end

    E -->|"success"| J["sink attached to engine.listeners"]
    E -->|"exception"| K["except Exception as error"]
    K --> L["logger.warning (stderr only)"]
    L --> M["ArtifactStore.write_json\nwandb/init-error.json"]
    M -->|"OSError"| N["logger.warning, continue"]
    M -->|"success"| O["wandb_sink = None\neval path continues"]
    N --> O

Loading

_{Reviews (4): Last reviewed commit: "fix: repair a scheme-less WANDB_BASE_URL..." | Re-trigger Greptile}

[shehabyasser-scale]

Confirmed live from inside a Modal sandbox

The open question on this PR was whether Modal egress to scaleai.wandb.io works, since the $0 repro only proved it from a laptop. It does.

A swe-atlas-qna run started at 08:58:16 on a branch carrying this fix. One minute later:

query_wandb_entity_projects(entity="shehab-yasser")
 -> [{"name": "vero-swe-atlas-qna", "created_at": "2026-07-25T05:59:21Z"}, ...]

That entity listed zero projects before today. The project was created by the eval-sidecar running inside the Modal sandbox, on its wandb.init().

And it is not just an empty project — it is streaming:

run vero-974cd10513e641f9   state: running   group: swe-atlas-qna
_step: 59        47 metric keys        has_history: true
inference/producer/requests: 129        inference/producer/total_tokens: 10,670,979
inference/evaluation/requests: 284      inference/evaluation/total_tokens: 245,524
validation/agent/num_cases: 48          validation/agent/metric/error_rate: 0.854
budget/harbor-validation/agent/remaining_runs: 5
diagnostics: "infrastructure_invalidity_threshold_exceeded"
status: "invalid"

Compare the failing run's state.json: evaluation_ids: [], next_step: 0, request_log_files: {}.

So the entire SidecarWandbSink pipeline — metrics, budget ledger, per-scope inference telemetry, diagnostics — was correct all along and was disabled by one missing https://.

Worth noting what this buys beyond the ticket: those inference/*/upstream_errors series are now readable mid-run, which is how I confirmed #54 and the model half of #58 without waiting for the run to finish or extracting a tarball. The observability fix immediately became the instrument for measuring the others.

Ready to merge from my side.

[varunursekar]

why not just fix the url input?