Repository navigation
feat(task-steps): native OSWorld initialize and evaluate on computer envs (FD-5492) - #606
williamloo-scale wants to merge 3 commits into
Conversation
…envs Adds the osworld_deploy_controller, osworld_initialize and osworld_evaluate steps and the osworld-server computer app (agent_env.osworld), so an OSWorld task (classic init_config/evaluator or an OSWorld-V2 task_NNN.py) runs on a framework computer env. - The OSWorld controller runs in a container sandbox of its own beside the computer, in remote mode, so the grader, a task's checks and the judge's key stay out of the agent's reach (a computer's apps run as its session user). - The computer runs only the OSWorld guest server, as the osworld-server app. Its card advertises urn:agentenv:osworld-guest/v1 at the prefix the app forwards to the guest server; the deploy step resolves the guest server's address from the computer's card (stored, else live until the app answers) and refuses a computer without the app before creating a sandbox. - osworld_initialize: OSWorld-V2 via /reset + /initialize_task; classic via /setup with the whole init_config. - osworld_evaluate: OSWorld-V2 via /evaluate; classic sends every func to the controller's /evaluate (async-polled, blocking fallback) and combines by conj; verifications shape matches cua_evaluate's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
williamloo-scale
changed the title
…tion test (FD-5492) A computer env with the osworld-server app, the controller beside it, a classic task set up and graded through it: pass scores 1.0, fail 0.0. Running it found: - The controller got the guest server's address as this process reaches it; a local container reaches this machine's loopback by another name. It now gets it, and the model endpoint, through url_from_sandbox, as an agent's deploy does. - supervisord looks a bare command up on PATH, so the guest server's service never started: it runs as `bash start-osworld-server.sh`. - python-xlib refuses to connect without an Xauthority file; the start script makes an empty one. - pyautogui's screenshots need scrot on X11; the app installs it. - A deployed env's record is stored once, when it is deployed, so the controller's address written on it lasted only the process. It is recorded in the run's context metadata too, and read from there. tst/integration/computer/osworld_test.py stages the app from OSWORLD_SERVER_SOURCE and runs the controller image OSWORLD_CONTROLLER_IMAGE (one with scale-cua #360); it skips without them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ility (FD-5492) It needs scale-cua's guest server source and a controller image, which this repository doesn't carry, so it skips without them. CI's skip policy fails a skip that isn't a declared capability gap, so the skip names osworld_sources, the slow tier allows it as it does mcp_server_sources, and CONTRIBUTING asks a change to the OSWorld code to run the test with both set and say so. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| sandbox = await provider.create_container(image_name=image.pinned_name, port=CONTROLLER_PORT, env=env, | ||
| cpu=self.cpu, memory=self.memory_mb, disk_size_gb=10, | ||
| timeout=self.ttl_seconds, attribution=deploy_attribution(self, context)) |
There was a problem hiding this comment.
On local Linux Docker, create_container publishes the controller's port on the Docker bridge gateway. A computer without a network restriction can reach that address. The controller accepts calls without authentication, so the agent can call /reset, /setup, or /evaluate directly and interfere with task state or grading. A separate container does not block this access.
Require a controller-only credential or put its listener behind a boundary the computer cannot reach.
How this was verified: The local provider publishes the controller port on the bridge gateway, the computer defaults to unrestricted networking, and the worker calls the controller routes without credentials.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/osworld/deploy_controller.py
Line: 102-104
Comment:
**Agent can reach the grader**
On local Linux Docker, `create_container` publishes the controller's port on the Docker bridge gateway. A computer without a network restriction can reach that address. The controller accepts calls without authentication, so the agent can call `/reset`, `/setup`, or `/evaluate` directly and interfere with task state or grading. A separate container does not block this access.
Require a controller-only credential or put its listener behind a boundary the computer cannot reach.
**How this was verified:** The local provider publishes the controller port on the bridge gateway, the computer defaults to unrestricted networking, and the worker calls the controller routes without credentials.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
| env = self.controller_env(deployed, provider.url_from_sandbox(guest_url)) | ||
| if "LITELLM_BASE_URL" in env: | ||
| env["LITELLM_BASE_URL"] = provider.url_from_sandbox(env["LITELLM_BASE_URL"]) |
There was a problem hiding this comment.
provider can still be a ChainedSandboxProvider, even when filtering leaves only one local link. That class inherits url_from_sandbox, which returns the URL unchanged. If the chain selects local Docker, a guest URL such as http://127.0.0.1:... points inside the controller instead of at the computer. A local LITELLM_BASE_URL fails the same way.
Build these values for the concrete provider being tried, not for the chain.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/osworld/deploy_controller.py
Line: 97-99
Comment:
**Fallback keeps the wrong address**
`provider` can still be a `ChainedSandboxProvider`, even when filtering leaves only one local link. That class inherits `url_from_sandbox`, which returns the URL unchanged. If the chain selects local Docker, a guest URL such as `http://127.0.0.1:...` points inside the controller instead of at the computer. A local `LITELLM_BASE_URL` fails the same way.
Build these values for the concrete provider being tried, not for the chain.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| n = len(funcs) | ||
| return list(zip(funcs, self.evaluator.get("result") or [None] * n, self.evaluator.get("expected") or [None] * n, | ||
| [o or {} for o in self.evaluator.get("options") or [{}] * n])) |
There was a problem hiding this comment.
zip silently drops trailing functions when a nonempty result, expected, or options list is shorter than funcs. For example, two functions with options: [{}] run only the first check and can record a passing score without checking the second.
Check list lengths and reject mismatches, or explicitly fill missing optional entries. This would keep task authors from accidentally leaving checks out of a grade.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/osworld/evaluate.py
Line: 162-164
Comment:
**Checks disappear from grades**
`zip` silently drops trailing functions when a nonempty `result`, `expected`, or `options` list is shorter than `funcs`. For example, two functions with `options: [{}]` run only the first check and can record a passing score without checking the second.
Check list lengths and reject mismatches, or explicitly fill missing optional entries. This would keep task authors from accidentally leaving checks out of a grade.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| try: | ||
| await run_setup(url, postconfig, what=f"osworld_evaluate '{self.verifier_id}' postconfig") | ||
| except Exception as e: | ||
| logger.warning(f"osworld_evaluate '{self.verifier_id}': postconfig failed, grading anyway: {e}") |
There was a problem hiding this comment.
When success is false and error is absent, run_setup puts the whole controller response into its exception. This warning then logs that response unchanged.
That violates the repository directive, “Never log full response bodies, request bodies, or large serialized objects in production code.” Log the failure type and a bounded summary instead. This repository requirement must be satisfied before merging.
Rule Used: What: Never log full response bodies, request bodies, or large serialized objects in production code. Why: Large logs are expensive, noisy, hard to search, risk pipeline limits, and can leak sensitive data. Good: ``` logger.info('Search completed',... (source)
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/osworld/evaluate.py
Line: 125
Comment:
**Setup responses enter logs**
When `success` is false and `error` is absent, `run_setup` puts the whole controller response into its exception. This warning then logs that response unchanged.
That violates the repository directive, “Never log full response bodies, request bodies, or large serialized objects in production code.” Log the failure type and a bounded summary instead. This repository requirement must be satisfied before merging.
**Rule Used:** What: Never log full response bodies, request bodies, or large serialized objects in production code. Why: Large logs are expensive, noisy, hard to search, risk pipeline limits, and can leak sensitive data. Good: ``` logger.info('Search completed',... ([source](https://app.greptile.com/-/switch-tenant?tenant=scale-ai&next=%2Fcustom-context%3Fmemory%3Dfa8d684f-4686-4f3e-b1ef-c27453f614ea))
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| logger.info(f"Deploying the OSWorld controller for env '{self.env_id}' (image={image.id} v{image.version}, " | ||
| f"env={sorted(env)})") | ||
| sandbox = await provider.create_container(image_name=image.pinned_name, port=CONTROLLER_PORT, env=env, | ||
| cpu=self.cpu, memory=self.memory_mb, disk_size_gb=10, |
There was a problem hiding this comment.
disk_size_gb=10 adds an unnamed controller resource setting. The repository requires magic numbers to be stored as class or instance variables with descriptive names. Give this disk size a named class setting so its purpose is clear and it can be changed in one place. This repository requirement must be satisfied before merging.
Rule Used: Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. (source)
Learned From
scaleapi/scaleapi#126388
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/osworld/deploy_controller.py
Line: 103
Comment:
**Disk setting lacks a name**
`disk_size_gb=10` adds an unnamed controller resource setting. The repository requires magic numbers to be stored as class or instance variables with descriptive names. Give this disk size a named class setting so its purpose is clear and it can be changed in one place. This repository requirement must be satisfied before merging.
**Rule Used:** Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. ([source](https://app.greptile.com/-/switch-tenant?tenant=scale-ai&next=%2Fcustom-context%3Fmemory%3D002e0051-41ad-46c1-9098-47433c580150))
**Learned From**
[scaleapi/scaleapi#126388](https://github.com/scaleapi/scaleapi/pull/126388)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Summary
This PR adds three task steps to core:
osworld_deploy_controller,osworld_initializeandosworld_evaluate. It also adds theosworld-servercomputer app (agent_env.osworld). Together they let an OSWorld task run on a frameworkcomputerenv. Both task kinds are supported:init_configplusevaluator.task_NNN.pythat definesTASK_CLASS.No plugin is needed.
Design
The controller runs as a separate sidecar.
The computer runs only the OSWorld guest server, as the
osworld-serverapp./osworld/<route>to127.0.0.1:5000.The guest server is found through the computer's card.
urn:agentenv:osworld-guest/v1at its forward prefix. AGETon that endpoint reports whether the guest server is up.osworld_deploy_controllerfinds the guest server by the extension's URI in the computer's card, and no/svc/<name>route is built anywhere.Step reference
osworld_deploy_controllerenv_id,controller_image_artifact_id(adocker_imageartifact; defaultcua-controller),ttl_seconds,cpu,memory_mb,env_vars.PROVIDER=remote,OS_TYPE=Ubuntu,REMOTE_VM_IP=<guest URL from the card>, andREMOTE_SERVER_HEADERS=<JSON of DeployedEnv.control_headers>. It also passesLITELLM_API_KEYandLITELLM_BASE_URLfrom[model]when they're configured.env_varsgo first, so the step's own values win.GET /configureanswers (the controller has no health route).DeployedEnv.metadata):osworld_controller_url,osworld_controller_sandbox_idandosworld_guest_url.context.deployed_sandboxesasosworld-controller-<env_id>.osworld_initializeenv_id,init_config,osworld_v2_task_path, and the legacyosworld_v2_task_url(the path wins if both are set).POST /reset, thenPOST /initialize_task {id, osworld_v2_task_url}. It fails onsuccess: falseor a truthyerror.init_configtoPOST /setup, which serves every setup type. It fails onsuccess: falseor on any seeded site withok: false. Seeded sites are recorded incontext.metadata["osworld_seeded_sites"].osworld_evaluateenv_id,evaluator,verifier_id(default: a uuid hex),osworld_v2_task_path/osworld_v2_task_url, andjudge_model./evaluate {osworld_v2_task_url}./evaluate, one at a time, with itsresultandexpectedgetters and its options. Results are combined byconj. No grading runs in the worker.infeasibleis the one exception: it is graded on the agent's answer (the[INFEASIBLE]tag), ascua_evaluatedoes.rubricandparameters, plusmodel=judge_modelwhen the func's options set none. Deterministic funcs have anymodeloption removed./evaluate_async, then polling/evaluate_result. A controller without those routes (404) gets the blocking/evaluate.stage_files(fromstep_overrides) is downloaded through/setupbefore anything reads the computer. The step refuses to grade if staging fails.postconfig: runs once through/setupbefore grading. A failure is logged and grading continues.error_typeis set is recorded as a skip with score 0. On classic tasks, an agent that ran out of turns or time is graded anyway and markedpartial_credit(on the verification and incontext.metadata["partial_credit"][verifier_id]).http(s)URL is passed through. An object-store ref is signed for 6 hours, but only if it's in the object store and under<prefix>artifacts/; anything else is refused.Result shape (same as
cua_evaluate)context.metadata["verifications"][verifier_id] = {"results": [...], "score": float}results=[{"id": "osworld_v2", "score", "result": not error, "message"}].results=[{"func", "results": [{"id", "score", "result", "message"}]}], plusdebug(overall_score,conjunction,metrics,error,message) andpartial_creditwhen it applies.Staging the app
The guest server's source isn't vendored; it's staged under
payload/. The app's declaration passes the checks a put runs. Then add the app to the computer env'sapps.Dependencies
http(s)://REMOTE_VM_IPbase URL that includes a path, and sendREMOTE_SERVER_HEADERSon every call to the guest server. This is scaleapi/scale-cua#360.collect_artifactson computer envs: feat(collect_artifacts): collect from framework computer envs #601 (separate PR).Known gaps
cua_evaluate's for the same task. These funcs are:compare_docx_files,compare_docx_tables,compare_pptx_files,compare_table,check_json_settings,check_json_keybindings,is_extension_installed,check_include_exclude,file_contains,run_sqlite3,check_thunderbird_prefsandcheck_thunderbird_filter.cua_evaluatereads (vm_file,cloud_file,rule,vm_command_line, the vscode getters) exists in its getters. So nothing becomes unservable.python3-pyatspi, which reaches the app's venv throughsys.path. This needs checking on a live computer.ubuntu-24.04.Testing
tst/unit/osworld/(guest resolution, controller transport, the three steps, the app)./evaluate_asyncpolling through transient errors, the 404 fallback, failed jobs, score normalization, and signing refused outside the artifacts.osworld_deploy_controller: env vars, metadata, the sandbox record, no secrets in the logs, refusal before any sandbox is created, and no double deploy.osworld_initialize: V2 and classic, including site failures.osworld_evaluate: V2, a V2 controller error scoring 0, the error_type skip for V2 and classic,conjand/or across funcs, a single func's partial score, a failing func among others, infeasible, partial credit, postconfig running once, andstage_files.ruff checkpasses.tst/unitandpackages/agentenv-protocol/tests): 14402 passed, 67 skipped, 3 failed. The 3 failures aretest_disk_recipe(Windows ISO boot) andtest_docker_image_github_clone(two cases of the token-signs-the-clone test); they fail the same way on main.Docs
The site should cover:
osworld-serverapp and theurn:agentenv:osworld-guest/v1extension;controller_image_artifact_id).End-to-end test
tst/integration/computer/osworld_test.pyruns on local Docker: a new computer env (osworld-desk: ubuntu-24.04, the reference driver, theosworld-serverapp), the controller built from scale-cua #360 beside it, a classicinit_configexecuteset up through the controller, thencheck_include_excludeovervm_command_linegraded by the controller. A rule that matches scores 1.0 and one that doesn't scores 0.0. It passes (2 m 33 s). It needsOSWORLD_SERVER_SOURCEandOSWORLD_CONTROLLER_IMAGEand skips without them.The first live run found four bugs, fixed in 460b019: the controller got a loopback address its container can't reach, the guest server's service command wasn't found, python-xlib needed an
~/.Xauthority, and screenshots neededscrot. The controller's address is also now saved in the run's context metadata, since a deployed env's record isn't re-saved after deploy.Not covered yet: an agent, an OSWorld-V2
.pytask, LLM-judge funcs, a cloud sandbox provider.🤖 Generated with Claude Code
Do not merge until the controller is protected from agent access, fallback URLs are corrected, and the repository requirements are satisfied.
Fix with agent prompt
Summary
Adds native OSWorld setup and grading on computer environments.
Acknowledged scope:
Diagram
sequenceDiagram participant Worker participant Controller participant Computer participant Guest Worker->>Computer: Read guest endpoint from card Worker->>Controller: Deploy with guest URL and control headers Worker->>Controller: Initialize or evaluate task Controller->>Computer: Guest request with control token Computer->>Guest: Forward request without token Guest-->>Controller: Task state or result Controller-->>Worker: Grade Note over Computer,Controller: Local Docker also lets computer code reach the controller portReviews (1) · Last reviewed commit: "test(osworld): the end-to-end test skips..." · Reviewed by Greptile