diff --git a/.github/workflows/deepgym-ci.yml b/.github/workflows/deepgym-ci.yml index 01a59304..0d76240e 100644 --- a/.github/workflows/deepgym-ci.yml +++ b/.github/workflows/deepgym-ci.yml @@ -36,8 +36,16 @@ jobs: cache: pip cache-dependency-path: deepgym/pyproject.toml + - name: Install bubblewrap (the local sandbox) + run: | + sudo apt-get update -q && sudo apt-get install -y -q bubblewrap + # Ubuntu 24.04 kernels with AppArmor block the unprivileged user namespaces bwrap needs. + knob=/proc/sys/kernel/apparmor_restrict_unprivileged_userns + if [ -e "$knob" ]; then sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0; fi + bwrap --ro-bind / / --unshare-all true + - name: Install dependencies - run: pip install -e ".[dev]" + run: pip install -e ".[dev,anthropic,openai]" - name: Lint (ruff) run: ruff check src/ tests/ @@ -64,6 +72,13 @@ jobs: cache: pip cache-dependency-path: deepgym/pyproject.toml + - name: Install bubblewrap (the local sandbox) + run: | + sudo apt-get update -q && sudo apt-get install -y -q bubblewrap + knob=/proc/sys/kernel/apparmor_restrict_unprivileged_userns + if [ -e "$knob" ]; then sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0; fi + bwrap --ro-bind / / --unshare-all true + # CPU-only torch keeps the job small. TRL 1.15 computes GRPO log-probs with Triton GPU # kernels, so CPU runners stay on TRL 1.14, whose reward and advantage code is the same. - name: Install dependencies diff --git a/deepgym/CHANGELOG.md b/deepgym/CHANGELOG.md index b297bb4d..d67f421f 100644 --- a/deepgym/CHANGELOG.md +++ b/deepgym/CHANGELOG.md @@ -1,5 +1,23 @@ # Changelog +## Unreleased + +- **Local sandbox reads only what a run needs:** under `sandbox-exec` and `bwrap`, a solution can read the operating system, the software trees a Python links against, the Python installation and its own run's files, and nothing else. Before, only the home directory, this process's runs and daemon sockets were hidden, so a solution could read files under `/tmp` and `$TMPDIR`, including another DeepGym process's verifier copies. Every deepgym package on the import path stays unreadable, not just the one running. Runs roots that a killed process left behind are removed by the next process. +- **macOS sandbox:** a solution can't signal anything outside its sandbox, its judge included. Nothing on macOS contains a child that detaches, so `run_solution(..., allow_subprocess=True)` is refused there (no verdict); run such environments on Linux or a sandbox provider. +- **A judge killed after the solution ran is its 0:** a local or sandboxed judge that printed no verdict after announcing the solution's start, or ended abnormally then (a signal, an exit outside 0, 1 and 2) whatever it printed before, scores the solution 0 (`solution_error`), as sessions already did, instead of no verdict. +- **No policy zero before the policy acts:** a session verifier that times out, loses its output or is killed before the agent's first action has no verdict; those were scored as the agent's 0. +- **Every agent, not only ours:** `run_episode` abandons the session for any agent that returns `provider_error` or `setup_error`, including one that builds its own `AgentResult`, and sets `truncated` on the grade of a run stopped at `output_limit`. +- **Cloud credentials refused:** E2B's `AuthenticationException` and Modal's `AuthError` and `PermissionDeniedError` count as infrastructure failures (no verdict), like a 401 or 403. +- **Python aliases under bwrap:** local sandboxes start the solution's Python by a path inside the installation (a `~/bin` or profile link goes by its target; a venv keeps its own directory). +- **Disk the agent filled:** a session verifier upload that fails after the agent acted is its judged 0 (`policy_sandbox_failure`), like any write, unless the network or provider failed (`VerifierUploadError`). +- **Docker and Apple transport:** a CLI that gives no answer, or a failed exec, while the daemon can't even list containers raises `SandboxUnreachableError` (infrastructure, no verdict) instead of grading the step as a timeout or an ordinary exit. While the daemon answers, it stays the command's outcome, since what ran in the container may have caused it. +- **No silent output cuts:** a script's stdout or stderr, or a file read with `read_output_bytes`, past its bound is the solution's failure instead of a prefix that could pass. A CLI run that prints past 256 MiB stops with the new `output_limit` termination reason (scored) instead of `error`. +- **cli_task verifier:** git objects count only if their content hashes to their name with the declared type and length, commits only if their tree exists and every entry of it parses as git's do, HEAD and `feature` must resolve to such commits, and the commit's tree must hold `hello.txt` with the right content. Forged ref, object and tree files no longer earn the git checks. +- **No sandbox, no local run:** where neither `sandbox-exec` nor `bwrap` works, local runs raise `SandboxError` instead of running with a warning. Set `DEEPGYM_ALLOW_UNSANDBOXED=1` to run unsandboxed anyway. +- **Provider failures get no verdict:** an agent episode that the model provider or the network ended (a cut or dropped stream, a 5xx, 429 or overloaded error, refused credentials) stops with `provider_error`, even an OpenAI response that failed with no usage. One whose harness never started the model (CLI missing or refused) stops with the new `setup_error`. Either way the session is abandoned (`Session.abandon`), so the grade has no verdict instead of a 0. Connection errors from the anthropic and openai SDKs' `httpx2` transport now count as infrastructure failures too. +- **CLI versions:** `ClaudeCodeAgent` and `CodexAgent` accept the tested CLI release or a later one in its major version (below 1.0, its minor version), instead of only the exact release. +- **Docs:** verifiers that can't judge must raise, not return 0. `ClaudeCodeAgent` drives the Claude Code CLI; there is no Claude Agent SDK integration. + ## 0.6.0 (2026-10-23) - **Agent harnesses:** `deepgym.agents` adds `AnthropicComputerAgent`, `OpenAIComputerAgent`, `ClaudeCodeAgent` and `CodexAgent`, each driving a live session. `DeepGym.run_episode(env, agent, budget=...)` runs the episode and returns `(AgentResult, RunResult)`; the callable interface still returns `(Trajectory, RunResult)`. The Claude Code and Codex CLIs are installed at pinned versions and checked in the sandbox. See `docs/agents.md`. diff --git a/deepgym/README.md b/deepgym/README.md index 24fd4e68..28f0cff2 100644 --- a/deepgym/README.md +++ b/deepgym/README.md @@ -295,8 +295,9 @@ scrolls and drags in screenshot pixels. More in [computer use][computer-use]. ### Run Claude Code or Codex in the sandbox -The CLI runs headless inside the session's sandbox as an unprivileged user, and DeepGym -scores the state it leaves: +`ClaudeCodeAgent` drives the Claude Code CLI (`claude -p`), not the Claude Agent SDK. The +CLI runs headless inside the session's sandbox as an unprivileged user, and DeepGym scores +the state it leaves: ```python from deepgym import DeepGym, MultiTurnEnvironment @@ -456,7 +457,7 @@ verl loads DeepGym's reward function with one config line (see | What it is | RL training framework | RL environments and verifiable rewards | | Does the training | Yes: rollouts on vLLM/SGLang, updates on FSDP/Megatron, PPO, GRPO, DAPO and more | No, it plugs into verl, TRL, OpenRLHF or your own loop | | Tasks | You bring the dataset and write the reward function | 31 built-in environments; importers for HumanEval, MBPP, EvalPlus, BigCodeBench, SWE-bench Pro, MiMo, CyberGym and Harbor datasets such as Terminal-Bench 2.0 | -| Running model code | Your reward function, or a remote code sandbox service | Daytona, Modal, E2B, Docker, Apple container, or locally under sandbox-exec/bwrap (without them, local runs get process separation and a warning) | +| Running model code | Your reward function, or a remote code sandbox service | Daytona, Modal, E2B, Docker, Apple container, or locally under sandbox-exec/bwrap (without them, local runs refuse unless `DEEPGYM_ALLOW_UNSANDBOXED=1`) | | Reward hacking | Up to your reward function | The solution runs apart from the verifier with no network, so it can't read the expected answers or print a fake score; `deepgym qa` checks that empty and stub solutions earn nothing and reward-hack ones stay below 0.5 before you train | | Multi-turn agents | Agent loop and tool interface | One live sandbox per episode (shell or desktop), with a verl agent loop and session tool | | Computer use | Not built in | Desktop tasks, scored on the state the agent leaves behind | diff --git a/deepgym/docs/agents.md b/deepgym/docs/agents.md index abd1822e..8e9bb594 100644 --- a/deepgym/docs/agents.md +++ b/deepgym/docs/agents.md @@ -5,7 +5,17 @@ `run(session, initial, *, task, budget) -> AgentResult`. Start the Session first; the caller owns grading and cleanup. `DeepGym.run_episode(env, agent, budget=...)` does that lifecycle and returns `(AgentResult, RunResult)`. Its existing callable -interface still returns `(Trajectory, RunResult)`. +interface still returns `(Trajectory, RunResult)`. `ClaudeCodeAgent` and `CodexAgent` +drive the Claude Code and Codex CLIs inside the sandbox; there is no Claude Agent SDK +integration. + +An episode that stops through nothing the policy did gets no verdict, never a 0: +`termination_reason` is `provider_error` when the model provider or the network failed (a +dropped or cut stream, a 5xx, 429 or overloaded error, our credentials refused) and +`setup_error` when the harness never started the model (its CLI missing or refused). The +session is then abandoned, so `run_episode`, or `score()` under `dg.trace()`, returns +`has_verdict=False` with that `error_type`, whatever state the policy reached. Every other +reason is scored, including a response the model itself cut at its output limit. Computer agents need the `anthropic` or `openai` optional extra, respectively. Imports stay lazy. Choose a model supporting Anthropic's @@ -141,9 +151,10 @@ its `stop_reason`, usage and `input_json`, each call's input exactly as it strea ## CLI isolation and its exposure CLIs must already be installed in the sandbox image; the library installs nothing. -Supported releases are pinned in `deepgym.agents.SUPPORTED_CLIS`. Each run checks -the installed version inside the sandbox before launching the task; unsupported -versions are refused. The observed version is retained in result and trace +The tested releases are in `deepgym.agents.SUPPORTED_CLIS`. Each run checks the +installed version inside the sandbox before launching the task: the tested release or a +later one in its major version (below 1.0, its minor version) runs; anything else is +refused as a `setup_error`. The observed version is retained in result and trace provenance. Their flags must match the adapters. The environment must allow `bash` or `cli`, and its backend must retain a separate network namespace. LocalBackend and provider-only network blocking without that namespace @@ -223,13 +234,14 @@ environments can retain it. ## Upgrading the CLIs `deepgym.agents.SUPPORTED_CLIS` is the sole source of supported versions and npm -package names. Phase 1 supports one exact release per CLI. Image builders can -generate the installation command without maintaining their own pins: +package names. The tested release is the oldest supported one; later compatible releases +run too. Image builders can generate the installation command for the tested releases +without maintaining their own pins: ```python from deepgym.agents import SUPPORTED_CLIS, cli_install_command -print(cli_install_command()) # npm install -g, with both packages pinned +print(cli_install_command()) # npm install -g, both tested releases print({name: release.version for name, release in SUPPORTED_CLIS.items()}) ``` diff --git a/deepgym/docs/authoring/verifiers.md b/deepgym/docs/authoring/verifiers.md index 56f484b6..e9976666 100644 --- a/deepgym/docs/authoring/verifiers.md +++ b/deepgym/docs/authoring/verifiers.md @@ -34,9 +34,12 @@ Never import or run the solution yourself. Use the helpers in `deepgym_solution` `subprocess.CompletedProcess`; check its `returncode`. Locally it can write only inside `cwd`; in a sandbox it runs as the solver user, which can still write world-writable paths. - Pass `allow_subprocess=True` if the script needs to start other programs (macOS blocks - forking otherwise), and `loopback_port=port` to let it reach a server the verifier runs - on `127.0.0.1:port` (nothing else on the network). + Pass `allow_subprocess=True` if the script needs to start other programs. Locally on + macOS that is refused (the run ends with no verdict), since nothing there contains a + child that detaches: run such environments on Linux or a sandbox provider. Pass + `loopback_port=port` to let it reach a server the verifier runs on `127.0.0.1:port` + (nothing else on the network). Output past 64 MiB, printed or read back with + `read_output_bytes`, is the solution's failure, never cut to a prefix. - `read_output(path, cwd)` reads its files back and refuses symlinks. - `solution_source(path, name=None)` reads the source for static checks without running it. @@ -51,6 +54,12 @@ verifier as a `deepgym_solution.SolutionError`, still of the builtin type the so error, so a solution can't dodge its score by breaking the verifier. Load test data before starting the solution: a verifier that crashes after that scores 0 too. +When the verifier itself can't judge (a fixture it needs is missing, a service it checks +against is down), it must raise, never return 0. Raised before it starts the solution, the +run ends with no verdict, which trainers mask; a returned 0 is a real verdict that training +learns from. So check what you need before starting the solution: after that, a crash is +the solution's 0, as above. + When the machine can't start the solution before any solution code has run (out of processes, say), these raise `LaunchError`. It isn't an `Exception`, so `except Exception` doesn't catch it. Let it propagate: the run ends with `sandbox_error` and no verdict. Once @@ -67,9 +76,12 @@ makes the whole run a judged 0. - **Sandboxes (Daytona, Modal, E2B, Docker, Apple container):** the solution runs as a separate Linux user that can't read the judge's files, touch the judge's process, or write to its output, with no network. -- **Locally:** it runs under `sandbox-exec` on macOS or `bwrap` on Linux, with the judge - directory, the deepgym package, your home directory and `/run` (daemon sockets) hidden. - If neither is available, you still get process separation and a warning. +- **Locally:** it runs under `sandbox-exec` on macOS or `bwrap` on Linux and can read only + the operating system, the software trees a Python links against (Homebrew's, say), the + Python installation and its own run's files. Your home directory, temp directories + (another run's verifier, an answer key under `/tmp`) and daemon sockets aren't + readable, and no deepgym package is. Where neither sandbox works, local runs refuse + unless you set `DEEPGYM_ALLOW_UNSANDBOXED=1`; the solution can then read your files. What this blocks: diff --git a/deepgym/docs/sandboxes.md b/deepgym/docs/sandboxes.md index d55d286f..c06d5512 100644 --- a/deepgym/docs/sandboxes.md +++ b/deepgym/docs/sandboxes.md @@ -78,9 +78,15 @@ one for anything shared or untrusted. (run any command as its sudoer) on loopback, so on both DeepGym gives untrusted code its own offline network namespace; that needs util-linux in the image, and a judge run fails rather than run online without it. -- **Locally:** the answer runs under `sandbox-exec` (macOS) or `bwrap` (Linux), with the - judge directory, the deepgym package, your home directory and `/run` (daemon sockets) - hidden. If neither is available, you still get process separation and a warning. +- **Locally:** the answer runs under `sandbox-exec` (macOS) or `bwrap` (Linux) and can + read only the operating system, the software trees a Python links against, the Python + installation and its own run's files: not your home directory, temp directories + (another run's verifier), daemon sockets or any deepgym package, and on macOS it can't + signal anything outside its sandbox, its judge included. macOS has no way to contain a + child that detaches, so there the answer never starts subprocesses and + `run_solution(..., allow_subprocess=True)` ends the run with no verdict: run such + environments on Linux or a sandbox provider. Where neither sandbox works, local runs + refuse unless you set `DEEPGYM_ALLOW_UNSANDBOXED=1`. `DeepGym(mode='local', confine_local_judge=True)` also runs each whole judge inside the local sandbox, for verifiers that might run code in-process; `deepgym qa --mutants` uses it. Sandboxes don't nest, so the answer then runs as plain child processes that diff --git a/deepgym/src/deepgym/_cli_worker.py b/deepgym/src/deepgym/_cli_worker.py index 02606a3a..c7384c3a 100644 --- a/deepgym/src/deepgym/_cli_worker.py +++ b/deepgym/src/deepgym/_cli_worker.py @@ -17,6 +17,10 @@ import time from pathlib import Path +# The CLI's output stream, which the host keeps whole for the trace and its events: a run +# that passes this is stopped as output_limit, its own termination reason, never an error. +# Far above a normal run's stream. +OUTPUT_LIMIT_BYTES = 256 * 1024 * 1024 _LOOPBACK_NO_PROXY = 'localhost,localhost.,127.0.0.1,::1,[::1],0:0:0:0:0:0:0:1,[0:0:0:0:0:0:0:1]' @@ -166,7 +170,7 @@ def _supervise(directory: Path) -> None: status['stop_reason'] = _stop_reason(directory, deadline) if status['stop_reason'] is not None: break - if output.tell() > 32 * 1024 * 1024: + if output.tell() > OUTPUT_LIMIT_BYTES: status['stop_reason'] = 'output_limit' break time.sleep(0.02) @@ -247,15 +251,25 @@ def _check_version(config: dict, status: dict, deadline: float) -> None: 'interpreter is executable on the same unprivileged PATH (for example ' '/usr/local/bin/node); do not load interactive nvm profiles.' ) - elif version != expected: + elif not cli_version_supported(version, expected): status['version_error'] = ( - f'Unsupported {command} version {version or "unknown"}; supported: {expected}. ' - 'Install the pinned release using deepgym.agents.cli_install_command(), ' - 'or bump SUPPORTED_CLIS, record versioned fixtures, run ' - 'tests/agents/test_cli_compat.py, and rebuild sandbox images.' + f'Unsupported {command} version {version or "unknown"}; supported: {expected} or ' + 'a later compatible release. Install one using ' + 'deepgym.agents.cli_install_command().' ) +def cli_version_supported(version: str | None, tested: str) -> bool: + """Whether *version* is *tested* or a later compatible release, which keeps the flags + and stream events the adapters use: the same major version, or below 1.0 the same + minor version, since there a minor release may break them.""" + if version is None: + return False + have, want = (tuple(map(int, release.split('.'))) for release in (version, tested)) + line = 1 if want[0] else 2 + return have[:line] == want[:line] and have >= want + + def _stop_reason(directory: Path, deadline: float) -> str | None: marker = directory / 'stop' if marker.exists(): diff --git a/deepgym/src/deepgym/agents/api.py b/deepgym/src/deepgym/agents/api.py index 86ec89b6..8c95d7cc 100644 --- a/deepgym/src/deepgym/agents/api.py +++ b/deepgym/src/deepgym/agents/api.py @@ -42,6 +42,7 @@ anthropic_unfinished_calls, api_error, openai_incomplete, + unresolved_usage, validate_budget, ) from deepgym.exceptions import ActionError, DeepGymError @@ -230,8 +231,9 @@ def run( timeout=tracker.request_timeout, ) episode.say('assistant', response) - if error := anthropic_incomplete(response): - return episode.result('error', error=error) + if incomplete := anthropic_incomplete(response): + reason, error = incomplete + return episode.result(reason, error=error) account(tracker, response.get('usage'), self.prices, 'anthropic') content = response['content'] messages.append({'role': 'assistant', 'content': content}) @@ -270,11 +272,11 @@ def run( except BudgetExceeded as exc: return episode.result(exc.reason) except DeepGymError as exc: - reason = 'time_budget' if tracker.remaining_seconds <= 0 else 'error' + reason = tracker.failure(exc) return episode.result(reason, error=f'Session step failed: {type(exc).__name__}') except anthropic.APIError as exc: tracker.add_usage(Usage(), Cost(provenance='Anthropic failed request; billing unknown')) - reason = 'time_budget' if tracker.remaining_seconds <= 0 else 'error' + reason = tracker.failure(exc) return episode.result(reason, error=f'Anthropic {api_error(exc)}') except (ValueError, KeyError, TypeError) as exc: return episode.result('error', error=f'Invalid Anthropic response: {exc}') @@ -355,6 +357,12 @@ def run( timeout=tracker.request_timeout, ).model_dump(exclude_none=True) episode.say('assistant', response) + incomplete = openai_incomplete(response) + if incomplete and incomplete[0] == 'provider_error': + # The provider's failure whatever else the response holds, which may be + # no usage at all: classified before accounting can fail on that. + unresolved_usage(tracker, response.get('usage'), 'openai') + return episode.result('provider_error', error=incomplete[1]) account(tracker, response.get('usage'), self.prices, 'openai') previous = response['id'] output = response['output'] @@ -368,8 +376,9 @@ def run( if refusal: return episode.result('refusal', refusal) text = ' '.join(part['text'] for part in parts if part['type'] == 'output_text') - if error := openai_incomplete(response): - return episode.result('error', error=error) + if incomplete: + reason, error = incomplete + return episode.result(reason, error=error) calls = [item for item in output if item['type'] == 'function_call'] if not calls: items = [{'role': 'user', 'content': _NO_CALL}] @@ -389,11 +398,11 @@ def run( except BudgetExceeded as exc: return episode.result(exc.reason) except DeepGymError as exc: - reason = 'time_budget' if tracker.remaining_seconds <= 0 else 'error' + reason = tracker.failure(exc) return episode.result(reason, error=f'Session step failed: {type(exc).__name__}') except openai.APIError as exc: tracker.add_usage(Usage(), Cost(provenance='OpenAI failed request; billing unknown')) - reason = 'time_budget' if tracker.remaining_seconds <= 0 else 'error' + reason = tracker.failure(exc) return episode.result(reason, error=f'OpenAI {api_error(exc)}') except (ValueError, KeyError, TypeError) as exc: return episode.result('error', error=f'Invalid OpenAI response: {exc}') diff --git a/deepgym/src/deepgym/agents/base.py b/deepgym/src/deepgym/agents/base.py index f0779dc4..e55888a9 100644 --- a/deepgym/src/deepgym/agents/base.py +++ b/deepgym/src/deepgym/agents/base.py @@ -8,7 +8,7 @@ from pydantic import BaseModel, ConfigDict, Field -from deepgym.exceptions import DeepGymError +from deepgym.exceptions import DeepGymError, is_infrastructure_failure from deepgym.models import Observation, Trajectory from deepgym.session import Session from deepgym.traces.schema import TraceMessage @@ -23,7 +23,34 @@ 'safety_check', 'error', 'provider_error', + 'setup_error', + 'output_limit', ] +"""Why an agent stopped. ``provider_error`` (the model or sandbox provider, or the network, +failed) and ``setup_error`` (the harness never started the model: its CLI missing or +refused) are nothing the policy did, so the episode gets no verdict (see +:meth:`BudgetTracker.result`). Every other reason is scored, ``output_limit`` too: a CLI +run stopped for printing more than the host keeps (see +:data:`deepgym._cli_worker.OUTPUT_LIMIT_BYTES`), shown as cut short rather than failed.""" + + +# Error types a provider names in an error body, mid-stream too (sent under the stream's 200). +_PROVIDER_ERROR_TYPES = frozenset( + {'api_error', 'overloaded_error', 'rate_limit_error', 'timeout_error', 'server_error'} +) + + +def provider_failed(exc: BaseException) -> bool: + """Whether *exc* is the provider's or the network's failure rather than the policy's. + + That is a failure :func:`~deepgym.exceptions.is_infrastructure_failure` names (transport, + 5xx, 429, our credentials refused), or a provider error body naming one of its own + failures (overloaded, say), which a stream reports mid-way under a 200 status. + """ + body = getattr(exc, 'body', None) + error = body.get('error', body) if isinstance(body, dict) else None + kind = error.get('type') if isinstance(error, dict) else None + return kind in _PROVIDER_ERROR_TYPES or is_infrastructure_failure(exc) class Budget(BaseModel): @@ -185,6 +212,13 @@ def add_usage(self, usage: Usage, cost: Cost) -> None: provenance='; '.join(source for source in sources if source != 'no provider calls'), ) + def failure(self, exc: BaseException) -> TerminationReason: + """Why an episode that *exc* ended stopped: ``time_budget`` once the clock ran out, + ``provider_error`` if :func:`provider_failed`, else ``error``.""" + if self.remaining_seconds <= 0: + return 'time_budget' + return 'provider_error' if provider_failed(exc) else 'error' + def result( self, session: Session, @@ -192,7 +226,13 @@ def result( final_text: str = '', error: str | None = None, ) -> AgentResult: - """Snapshot accounting and the session's real actions without grading.""" + """Snapshot accounting and the session's real actions without grading. + + A reason that is nothing the policy did (``provider_error``, ``setup_error``) + also abandons the session, so whoever scores it gets no verdict, not a 0. + """ + if reason in ('provider_error', 'setup_error'): + session.abandon(reason, error or reason) return AgentResult( trajectory=session.trajectory.model_copy(deep=True), usage=self.usage, diff --git a/deepgym/src/deepgym/agents/cli.py b/deepgym/src/deepgym/agents/cli.py index 429d4d25..b6f22034 100644 --- a/deepgym/src/deepgym/agents/cli.py +++ b/deepgym/src/deepgym/agents/cli.py @@ -113,7 +113,7 @@ def _run(self, session: Session, task: str, budget: Budget) -> AgentResult: + '; CLI failure: usage may be incomplete', } ) - reason = 'time_budget' if tracker.remaining_seconds <= 0 else 'error' + reason = tracker.failure(exc) result = tracker.result(session, reason, events.final_text, type(exc).__name__) if isinstance(exc, CliExecutionError): result = result.model_copy(update={'cli_version': exc.cli_version}) @@ -123,7 +123,7 @@ def _run(self, session: Session, task: str, budget: Budget) -> AgentResult: reason = ( 'time_budget' if execution.stop_reason == 'time_budget' or tracker.remaining_seconds <= 0 - else 'error' + else 'setup_error' ) result = tracker.result(session, reason, error=execution.version_error) else: diff --git a/deepgym/src/deepgym/agents/cli_events.py b/deepgym/src/deepgym/agents/cli_events.py index 41c4092d..aea233dd 100644 --- a/deepgym/src/deepgym/agents/cli_events.py +++ b/deepgym/src/deepgym/agents/cli_events.py @@ -166,7 +166,8 @@ def finish(self, exit_code: int, stop_reason: str | None) -> TerminationReason: The observed episode termination reason. """ reason = self.stop_reason or stop_reason - if reason not in ('turn_budget', 'time_budget', 'cost_budget', 'provider_error', 'error'): + passthrough = ('turn_budget', 'time_budget', 'cost_budget', 'output_limit') + if reason not in (*passthrough, 'provider_error', 'error'): if self.error is None and exit_code != 0: self.error = f'{self.provider} CLI exited with code {exit_code}' if self.error is None and not self.completed: diff --git a/deepgym/src/deepgym/agents/computer.py b/deepgym/src/deepgym/agents/computer.py index 2838219f..6f2a7202 100644 --- a/deepgym/src/deepgym/agents/computer.py +++ b/deepgym/src/deepgym/agents/computer.py @@ -30,6 +30,7 @@ anthropic_unfinished_calls, api_error, openai_incomplete, + unresolved_usage, validate_budget, ) from deepgym.exceptions import ActionError, DeepGymError @@ -142,8 +143,9 @@ def run( messages=messages, timeout=tracker.request_timeout, ) - if error := anthropic_incomplete(response): - return tracker.result(session, 'error', error=error) + if incomplete := anthropic_incomplete(response): + reason, error = incomplete + return tracker.result(session, reason, error=error) account(tracker, response.get('usage'), self.prices, 'anthropic') content = response['content'] messages.append({'role': 'assistant', 'content': content}) @@ -204,13 +206,13 @@ def run( except BudgetExceeded as exc: return tracker.result(session, exc.reason) except DeepGymError as exc: - reason = 'time_budget' if tracker.remaining_seconds <= 0 else 'error' + reason = tracker.failure(exc) return tracker.result( session, reason, error=f'Session step failed: {type(exc).__name__}' ) except anthropic.APIError as exc: tracker.add_usage(Usage(), Cost(provenance='Anthropic failed request; billing unknown')) - reason = 'time_budget' if tracker.remaining_seconds <= 0 else 'error' + reason = tracker.failure(exc) return tracker.result(session, reason, error=f'Anthropic {api_error(exc)}') except (ValueError, KeyError, TypeError) as exc: return tracker.result(session, 'error', error=f'Invalid Anthropic response: {exc}') @@ -277,6 +279,12 @@ def run( **({'reasoning': {'effort': self.effort}} if self.effort else {}), timeout=tracker.request_timeout, ).model_dump(exclude_none=True) + incomplete = openai_incomplete(response) + if incomplete and incomplete[0] == 'provider_error': + # The provider's failure whatever else the response holds, which may be + # no usage at all: classified before accounting can fail on that. + unresolved_usage(tracker, response.get('usage'), 'openai') + return tracker.result(session, 'provider_error', error=incomplete[1]) account(tracker, response.get('usage'), self.prices, 'openai') previous = response['id'] output = response['output'] @@ -297,8 +305,9 @@ def run( 'safety_check', error=('Provider safety checks require approval; no actions executed'), ) - if error := openai_incomplete(response): - return tracker.result(session, 'error', error=error) + if incomplete: + reason, error = incomplete + return tracker.result(session, reason, error=error) if not calls: return tracker.result(session, 'completed', text) items = [] @@ -338,13 +347,13 @@ def run( except BudgetExceeded as exc: return tracker.result(session, exc.reason) except DeepGymError as exc: - reason = 'time_budget' if tracker.remaining_seconds <= 0 else 'error' + reason = tracker.failure(exc) return tracker.result( session, reason, error=f'Session step failed: {type(exc).__name__}' ) except openai.APIError as exc: tracker.add_usage(Usage(), Cost(provenance='OpenAI failed request; billing unknown')) - reason = 'time_budget' if tracker.remaining_seconds <= 0 else 'error' + reason = tracker.failure(exc) return tracker.result(session, reason, error=f'OpenAI {api_error(exc)}') except (ValueError, KeyError, TypeError) as exc: return tracker.result(session, 'error', error=f'Invalid OpenAI response: {exc}') diff --git a/deepgym/src/deepgym/agents/providers.py b/deepgym/src/deepgym/agents/providers.py index 032ffe0c..6837105b 100644 --- a/deepgym/src/deepgym/agents/providers.py +++ b/deepgym/src/deepgym/agents/providers.py @@ -8,9 +8,18 @@ from __future__ import annotations +import contextlib from typing import Any, Literal -from deepgym.agents.base import Budget, BudgetTracker, Cost, TokenPrices, Usage +from deepgym.agents.base import ( + Budget, + BudgetTracker, + Cost, + TerminationReason, + TokenPrices, + Usage, + provider_failed, +) # The SDK retries connection drops, 429s and 5xx with backoff; one blip shouldn't end an # hours-long episode. A retried request may be billed twice; usage counts only answers. @@ -28,13 +37,16 @@ def _snapshot(stream: Any) -> Any: return None -def _incomplete( - tracker: BudgetTracker, message: dict[str, Any], prices: TokenPrices | None +def unresolved_usage( + tracker: BudgetTracker, data: object, provider: Literal['anthropic', 'openai'] ) -> None: - """Keep the usage an unfinished stream reported, with the total bill marked unresolved.""" - data = message.get('usage') - usage = _anthropic_usage(data) if isinstance(data, dict) else Usage() - unresolved = 'incomplete streamed response: partial usage, billing unresolved' + """Keep the usage an unfinished response reported, if any (a failed one may carry + none), with the total bill marked unresolved.""" + usage = Usage() + if isinstance(data, dict): + with contextlib.suppress(KeyError, TypeError, ValueError): + usage = _anthropic_usage(data) if provider == 'anthropic' else _openai_usage(data) + unresolved = f'unfinished {provider} response: partial usage, billing unresolved' tracker.add_usage(usage, Cost(provenance=unresolved)) @@ -80,24 +92,29 @@ def anthropic_message( if fragments and message.get('stop_reason') != 'tool_use': message['input_json'] = fragments if failure is not None or not stopped or snapshot is None: - _incomplete(tracker, message, prices) - details = {'reason': 'Response stream ended before completion'} + unresolved_usage(tracker, message.get('usage'), 'anthropic') + # A stream that just ended was cut on the way; one that failed is the provider's + # failure only if it names one (a delta the SDK can't parse is the model's output). + details = {'reason': 'Response stream ended before completion', 'provider': True} if failure is not None: # Its text too: the SDK's error for a delta it can't parse quotes that delta, # which never reaches the snapshot or the fragments above. details = { 'reason': f'Response stream failed: {type(failure).__name__}', 'detail': str(failure), + 'provider': provider_failed(failure), } return {**message, 'status': 'incomplete', 'incomplete_details': details} return message -def anthropic_incomplete(message: dict[str, Any]) -> str | None: - """Why :func:`anthropic_message` returned *message* incomplete, or None if it completed.""" +def anthropic_incomplete(message: dict[str, Any]) -> tuple[TerminationReason, str] | None: + """How and why :func:`anthropic_message` returned *message* incomplete, or None if it + completed: ``provider_error`` when the provider or network cut it, else ``error``.""" if message.get('status') != 'incomplete': return None - return message['incomplete_details']['reason'] + details = message['incomplete_details'] + return 'provider_error' if details['provider'] else 'error', details['reason'] def anthropic_unfinished_calls(message: dict[str, Any]) -> str | None: @@ -112,16 +129,20 @@ def anthropic_unfinished_calls(message: dict[str, Any]) -> str | None: return f'Response stopped ({stop}) before its tool calls were complete' -def openai_incomplete(response: dict[str, Any]) -> str | None: - """Why an OpenAI response did not complete, or None if it did. +def openai_incomplete(response: dict[str, Any]) -> tuple[TerminationReason, str] | None: + """How and why an OpenAI response did not complete, or None if it did. - Never act on, or blindly continue, a partial response; keep its reason. + Never act on, or blindly continue, a partial response; keep its reason. One + ``incomplete`` stopped at the model's own output limit or a content filter: the + policy's (``error``). Any other status (failed, cancelled) is the provider's + (``provider_error``). """ if response.get('status') == 'completed': return None details = response.get('incomplete_details') or {} reason = details.get('reason') or response.get('status') - return f'Response did not complete: {reason}' + policy = response.get('status') == 'incomplete' + return 'error' if policy else 'provider_error', f'Response did not complete: {reason}' def account( diff --git a/deepgym/src/deepgym/agents/versions.py b/deepgym/src/deepgym/agents/versions.py index 8d31f62c..cb4e9c0b 100644 --- a/deepgym/src/deepgym/agents/versions.py +++ b/deepgym/src/deepgym/agents/versions.py @@ -1,4 +1,4 @@ -"""Pin supported CLI releases in one place for adapters and image builders.""" +"""Name the supported CLI releases in one place for adapters and image builders.""" import shlex from collections.abc import Mapping @@ -9,7 +9,8 @@ class CliVersion(BaseModel): - """Describe one tested release; the supported range is this exact version.""" + """Describe the tested release: the oldest supported, and the one image builders install. + Later releases in its major version are accepted too.""" model_config = ConfigDict(frozen=True) command: str @@ -28,7 +29,7 @@ class CliVersion(BaseModel): def cli_install_command() -> str: - """Return the pinned npm installation command for sandbox image builders. + """Return the npm installation command for the tested releases, for image builders. It installs under /usr/local, so the binaries land on the standard PATH the sandbox agent gets even when the image's Node comes from nvm or another custom prefix. diff --git a/deepgym/src/deepgym/async_core.py b/deepgym/src/deepgym/async_core.py index 454fe5ed..0528a634 100644 --- a/deepgym/src/deepgym/async_core.py +++ b/deepgym/src/deepgym/async_core.py @@ -421,7 +421,7 @@ async def _run_judge(self, sandbox: Sandbox, job: JudgeJob) -> ExecutionResult: # Daytona returns nothing the verifier printed, so even a sandbox that died # can't be shown to have died after the solution started: no verdict. raise SandboxError(f'Verifier execution failed: {exc}') from exc - return parse_wrapped_output(response.result, marker, response.exit_code) + return parse_wrapped_output(response.result, marker, response.exit_code, job.consulted_mark) async def _cleanup_sandbox(self, sandbox: Sandbox) -> None: """Safely destroy a sandbox, swallowing errors.""" diff --git a/deepgym/src/deepgym/cli.py b/deepgym/src/deepgym/cli.py index 9c468fd5..5f66358f 100644 --- a/deepgym/src/deepgym/cli.py +++ b/deepgym/src/deepgym/cli.py @@ -14,7 +14,7 @@ from typing import TYPE_CHECKING from deepgym.factory.scaffold import KINDS -from deepgym.judge import local_isolation_kind +from deepgym.judge import ALLOW_UNSANDBOXED_ENV, local_isolation_kind, unsandboxed_allowed from deepgym.sandboxes import PROVIDERS if TYPE_CHECKING: @@ -560,8 +560,9 @@ def _cmd_check(args: argparse.Namespace) -> None: def _guard_local_model_code(mode: str, allowed: bool) -> None: """Exit unless the user opted in to running model-generated code on this machine. - 'auto' counts as local because it silently falls back to local execution. - With the opt-in but no sandbox-exec or bwrap, print a warning and continue. + 'auto' counts as local because it silently falls back to local execution. With the + opt-in but no sandbox-exec or bwrap, exit too, unless the user also allowed running + unsandboxed (:data:`~deepgym.judge.ALLOW_UNSANDBOXED_ENV`); then warn and continue. """ if not allowed: print( @@ -572,6 +573,14 @@ def _guard_local_model_code(mode: str, allowed: bool) -> None: ) sys.exit(1) if local_isolation_kind() == 'none': + if not unsandboxed_allowed(): + print( + 'Error: no working sandbox-exec or bwrap on this machine, so model-generated ' + 'code would run unsandboxed and could read your files. Install bubblewrap, ' + f'use a sandboxed mode, or set {ALLOW_UNSANDBOXED_ENV}=1 to accept the risk.', + file=sys.stderr, + ) + sys.exit(1) print( 'WARNING: no sandbox-exec or bwrap on this machine. Model-generated code will run ' 'with only process separation and can read your files.', diff --git a/deepgym/src/deepgym/core.py b/deepgym/src/deepgym/core.py index 07c5b24a..566e0ce4 100644 --- a/deepgym/src/deepgym/core.py +++ b/deepgym/src/deepgym/core.py @@ -235,8 +235,9 @@ def run_episode( Returns: Tuple of (AgentResult, final_result) for a harness, or - (trajectory, final_result) for a legacy callable. The grade describes - final state independently of the harness termination reason. + (trajectory, final_result) for a legacy callable. An agent that stopped through + nothing the policy did (``provider_error``, ``setup_error``) gets no verdict; + one the harness cut short (``output_limit``) is graded with ``truncated`` set. Raises: ValueError: If a budget is supplied to a legacy callable. @@ -247,7 +248,13 @@ def run_episode( with self.session(env) as session: initial = session.start() result = agent.run(session, initial, task=env.task, budget=budget or Budget()) - return result, session.score() + reason = result.termination_reason + if reason in ('provider_error', 'setup_error'): # any Agent, not only ours + session.abandon(reason, result.error or reason) + grade = session.score() + if reason == 'output_limit': # the harness cut the run short + grade = grade.model_copy(update={'truncated': True}) + return result, grade if budget is not None: raise ValueError('budget requires an Agent harness with a run method') return run_episode(self.session(env), agent) diff --git a/deepgym/src/deepgym/envs/cli_task/verifier.py b/deepgym/src/deepgym/envs/cli_task/verifier.py index dec42f9a..dd192ef8 100644 --- a/deepgym/src/deepgym/envs/cli_task/verifier.py +++ b/deepgym/src/deepgym/envs/cli_task/verifier.py @@ -1,5 +1,6 @@ """Verifier for the CLI Task (Git Operations).""" +import hashlib import json import os import re @@ -10,58 +11,136 @@ from deepgym_solution import read_output, read_output_bytes, run_solution +_SHA = re.compile(r'[0-9a-f]{40}') +# The most of one file the judge holds in memory, as deepgym_solution's read_output_bytes: +# an object that inflates past it isn't read at all (a zip bomb would exhaust the judge). +_JUDGE_HOLDS = 64 * 1024 * 1024 -def _read_repo(repo_dir, root): - """Return (branch names, commit messages) read straight from .git. - Never runs git: a repo the solution built can set .git/config options that - make git execute commands. Reads only loose objects, through read_output_bytes - (no symlinks, nothing outside *root*). +class _Repo: + """A git repository read straight from .git, trusting only what checks out. + + Never runs git: a repo the solution built can set .git/config options that make git + execute commands. Reads only loose objects, through read_output_bytes (no symlinks, + nothing outside *root*), and takes an object only if its content hashes to its name + with the type and length its header declares, so nothing can be forged by writing + files that merely look like git's. """ - git_dir = os.path.join(repo_dir, '.git') - def read(rel): + def __init__(self, repo_dir, root): + self.git_dir = os.path.join(repo_dir, '.git') + self.root = root + packed = (self._read('packed-refs') or b'').decode('utf-8', 'replace') + self.packed = {} + for line in packed.splitlines(): + sha, _, name = line.partition(' ') + if not line.startswith(('#', '^')): + self.packed[name.strip()] = sha.strip() + + def _read(self, rel): try: - return read_output_bytes(os.path.join(git_dir, rel), root, limit=10_000_000) - except OSError: + return read_output_bytes(os.path.join(self.git_dir, rel), self.root) + except OSError: # missing, a link, outside root, or more than the judge holds return None - branches = {} - packed = read('packed-refs') or b'' - for line in packed.decode('utf-8', 'replace').splitlines(): - sha, _, name = line.partition(' ') - if name.startswith('refs/heads/'): - branches[name[len('refs/heads/') :]] = sha.strip() - heads = os.path.join(git_dir, 'refs', 'heads') - try: - names = [] if os.path.islink(heads) else os.listdir(heads) - except OSError: # missing, or made unreadable by the solution - names = [] - for name in names: - sha = read(os.path.join('refs', 'heads', name)) - if sha is not None: - branches[name] = sha.decode('ascii', 'replace').strip() - - messages, seen, todo = [], set(), list(branches.values()) - while todo and len(seen) < 1000: - sha = todo.pop() - if sha in seen or not re.fullmatch(r'[0-9a-f]{40}', sha): - continue - seen.add(sha) - compressed = read(os.path.join('objects', sha[:2], sha[2:])) + def ref(self, name): + """The commit id *name* (refs/heads/...) points to, or None.""" + loose = self._read(name) + sha = ( + loose.decode('ascii', 'replace').strip() if loose is not None else self.packed.get(name) + ) + return sha if sha and _SHA.fullmatch(sha) else None + + def head(self): + """The commit id HEAD resolves to, through its branch or detached, or None.""" + head = (self._read('HEAD') or b'').decode('utf-8', 'replace').strip() + if head.startswith('ref: refs/heads/'): + return self.ref(head[len('ref: ') :]) + return head if _SHA.fullmatch(head) else None + + def branches(self): + """Branch name to commit id, loose refs over packed ones.""" + names = { + name[len('refs/heads/') :] for name in self.packed if name.startswith('refs/heads/') + } + heads = os.path.join(self.git_dir, 'refs', 'heads') + try: + names |= set() if os.path.islink(heads) else set(os.listdir(heads)) + except OSError: # missing, or made unreadable by the solution + pass + refs = {name: self.ref(f'refs/heads/{name}') for name in names} + return {name: sha for name, sha in refs.items() if sha} + + def object(self, sha, kind): + """The body of loose object *sha* if it really is that *kind* of object, else None.""" + if not sha or not _SHA.fullmatch(sha): + return None + compressed = self._read(os.path.join('objects', sha[:2], sha[2:])) if compressed is None: - continue + return None + inflater = zlib.decompressobj() try: - raw = zlib.decompressobj().decompress(compressed, 1_000_000) + raw = inflater.decompress(compressed, _JUDGE_HOLDS) except zlib.error: - continue + return None + if inflater.unconsumed_tail or not inflater.eof or inflater.unused_data: + return None # more than the judge holds, cut short, or trailing bytes + if hashlib.sha1(raw).hexdigest() != sha: + return None header, _, body = raw.partition(b'\0') - if not header.startswith(b'commit '): - continue - meta, _, message = body.decode('utf-8', 'replace').partition('\n\n') - messages.append(message.strip()) - todo += [line[len('parent ') :] for line in meta.splitlines() if line.startswith('parent ')] - return set(branches), messages + return body if header == f'{kind} {len(body)}'.encode() else None + + def commit(self, sha): + """(tree id, parent ids, message) of commit *sha*, or None unless it is one whose + tree is a real, well-formed tree.""" + body = self.object(sha, 'commit') + if body is None: + return None + meta, _, message = body.partition(b'\n\n') + lines = meta.decode('utf-8', 'replace').split('\n') + tree = lines[0][len('tree ') :] + if not lines[0].startswith('tree ') or self.entries(tree) is None: + return None + parents = [line[len('parent ') :] for line in lines[1:] if line.startswith('parent ')] + return tree, parents, message.decode('utf-8', 'replace') + + def entries(self, tree): + """{name: (mode, object id)} of *tree*, or None unless every entry is well formed: + an octal mode, a name, a NUL and a 20-byte id, as git itself parses it.""" + body = self.object(tree, 'tree') + if body is None: + return None + found = {} + while body: + entry, sep, rest = body.partition(b'\0') + mode, space, name = entry.partition(b' ') + if not sep or not space or not name or len(rest) < 20 or b'/' in name: + return None + if not mode or any(c not in b'01234567' for c in mode) or name in found: + return None + found[name] = (mode, rest[:20].hex()) + body = rest[20:] + return found + + def file_in(self, tree, name): + """The content of regular file *name* at the top of well-formed *tree*, or None.""" + mode, sha = (self.entries(tree) or {}).get(name.encode(), (None, None)) + return self.object(sha, 'blob') if mode in (b'100644', b'100755') else None + + def commits(self, starts): + """(message, tree id) of every valid commit reachable from *starts*.""" + found, seen, todo = [], set(), list(starts) + while todo: + sha = todo.pop() + if sha in seen: + continue + seen.add(sha) + commit = self.commit(sha) + if commit is not None: + tree, parents, message = commit + found.append((message, tree)) + todo += parents + return found def verify(solution_path, test_cases_path=None): @@ -85,13 +164,18 @@ def verify(solution_path, test_cases_path=None): # Check repo directory exists. checks['repo_exists'] = os.path.isdir(repo_dir) - # Check it is a git repo, and read its branches and commits without running git. + # Check it is a git repo whose HEAD resolves to a real commit, read without running git. git_dir = os.path.join(repo_dir, '.git') - checks['is_git_repo'] = os.path.isdir(git_dir) and not os.path.islink(git_dir) - branches, messages = _read_repo(repo_dir, work_dir) if checks['is_git_repo'] else (set(), []) + repo = _Repo(repo_dir, work_dir) if os.path.isdir(git_dir) else None + if repo is not None and os.path.islink(git_dir): + repo = None + checks['is_git_repo'] = repo is not None and repo.commit(repo.head()) is not None + branches = repo.branches() if repo is not None else {} - # Check 'feature' branch exists. - checks['feature_branch_exists'] = 'feature' in branches + # Check 'feature' branch exists and points to a real commit. + checks['feature_branch_exists'] = ( + repo is not None and repo.commit(branches.get('feature')) is not None + ) # Check hello.txt exists with correct content. try: @@ -102,8 +186,12 @@ def verify(solution_path, test_cases_path=None): checks['hello_file_exists'] = False checks['hello_content_correct'] = False - # Check a commit has the expected message. - checks['has_commit'] = any(m.startswith('Add hello.txt') for m in messages) + # Check a real commit has the expected message and commits the file itself. + checks['has_commit'] = repo is not None and any( + message.startswith('Add hello.txt') + and (repo.file_in(tree, 'hello.txt') or b'').strip() == b'Hello, DeepGym!' + for message, tree in repo.commits(branches.values()) + ) # Check solution prints 'done'. checks['prints_done'] = 'done' in proc.stdout.strip().lower() diff --git a/deepgym/src/deepgym/exceptions.py b/deepgym/src/deepgym/exceptions.py index 705ecf8a..5a185b8f 100644 --- a/deepgym/src/deepgym/exceptions.py +++ b/deepgym/src/deepgym/exceptions.py @@ -9,6 +9,9 @@ _TRANSPORT_ERRORS = ( ('httpx', ('TransportError',)), + ('httpx2', ('TransportError',)), # the Anthropic and OpenAI SDKs' transport, newer releases + ('anthropic', ('APIConnectionError',)), + ('openai', ('APIConnectionError',)), ( 'urllib3.exceptions', ('ProtocolError', 'NewConnectionError', 'MaxRetryError', 'TimeoutError'), @@ -18,7 +21,12 @@ ('daytona', ('DaytonaConnectionError', 'DaytonaSpotEvictedError', 'DaytonaQueueTimeoutError')), ( 'e2b.exceptions', - ('SandboxUnreachableException', 'ServiceBusyException', 'RateLimitException'), + ( + 'SandboxUnreachableException', + 'ServiceBusyException', + 'RateLimitException', + 'AuthenticationException', # our credentials refused; it carries no status + ), ), ('pyqwest', ('ReadError', 'WriteError', 'StreamError')), # e2b's RPC transport ( @@ -29,6 +37,8 @@ 'InternalFailure', 'ResourceExhaustedError', 'ServiceError', + 'AuthError', # our credentials refused; Modal's status is an Enum, not an int + 'PermissionDeniedError', ), ), ) @@ -74,6 +84,12 @@ def __init__(self, message: str, output: str = '') -> None: self.output = output +class SandboxUnreachableError(SandboxError): + """Raise when the provider gave no answer about a command, or none to trust: its daemon + hung or can't be reached, before the command started or while it ran. Infrastructure, + never the policy's doing (see :func:`is_infrastructure_failure`).""" + + class SandboxExpiredError(SandboxError): """Raise when the provider retired the sandbox at the end of its lifetime, whatever ran in it. Not the policy's doing unless its own actions ran the clock out (see @@ -136,7 +152,9 @@ def is_infrastructure_failure(exc: BaseException) -> bool: while exc is not None: if isinstance(exc, (TimeoutError, SandboxDiedError, *gone)): return False - if isinstance(exc, (ConnectionError, builtins.TimeoutError, *transport)): + if isinstance(exc, (ConnectionError, builtins.TimeoutError, SandboxUnreachableError)): + return True + if isinstance(exc, transport): return True status = _status_code(exc) if status is not None and (status >= 500 or status in (401, 403, 429)): diff --git a/deepgym/src/deepgym/judge.py b/deepgym/src/deepgym/judge.py index c32bb664..f899de5e 100644 --- a/deepgym/src/deepgym/judge.py +++ b/deepgym/src/deepgym/judge.py @@ -180,52 +180,166 @@ def local_isolation_kind() -> Literal['macos', 'bwrap', 'none']: with contextlib.suppress(OSError, subprocess.SubprocessError): if subprocess.run(probe, capture_output=True, timeout=10).returncode == 0: return kind # type: ignore[return-value] - logger.warning( - 'No sandbox-exec or bwrap available: local solutions run in a separate process but ' - 'can read judge files. Use Daytona for untrusted code.' - ) return 'none' +ALLOW_UNSANDBOXED_ENV = 'DEEPGYM_ALLOW_UNSANDBOXED' +"""Set to 1 to run local solutions with no sandbox, where none works; they can then read +every file this user can, verifiers and answer keys included.""" + + +def unsandboxed_allowed() -> bool: + """Whether the user opted in to local runs with no sandbox (:data:`ALLOW_UNSANDBOXED_ENV`).""" + return os.environ.get(ALLOW_UNSANDBOXED_ENV, '').lower() in ('1', 'true', 'yes') + + +# What a local solution may read besides the Python installation and its own run: the +# operating system and the software trees a Python links against (Homebrew's, MacPorts', +# Nix's, Apple's developer tools). Nothing else is readable, so nothing a user or another +# process wrote is: home, temp directories (other runs and their verifiers), mounted +# volumes, daemon sockets in /run and /var/run. +_SYSTEM_READ = { + 'darwin': ( + '/System', + '/usr', + '/bin', + '/sbin', + '/dev', + '/private/etc', + '/private/var/db/timezone', + '/private/var/select', + '/private/var/db/xcode_select_link', + '/Library/Frameworks', + '/Library/Developer/CommandLineTools', + '/Applications/Xcode.app/Contents/Developer', + '/opt/homebrew', + '/opt/local', + '/nix/store', + ), + 'linux': ( + '/usr', + '/bin', + '/sbin', + '/lib', + '/lib32', + '/lib64', + '/libx32', + '/etc', + '/sys', + '/nix/store', + '/gnu/store', + ), +} +# Service data inside those trees (databases, logs), hidden again. +_SYSTEM_HIDE = ('/opt/homebrew/var', '/opt/local/var', '/usr/local/var') + + _runs_root_lock = threading.Lock() _runs_root: list[str] = [] +_RUNS_PREFIX = 'deepgym_runs_' +_RUNS_LOCK = '.owner.lock' def local_runs_root() -> str: - """One parent for every local run in this process, hidden from all solutions. + """One parent for every local run in this process, unreadable to all solutions. Anything else no local solution may read (an approval's snapshot) goes here too. + It is removed at exit; this process holds a lock in it for life, so a later process + can tell a root left by one that was killed and remove it (see :func:`_remove_stale_roots`). - Locked: parallel batch runs must share it, or each would hide only its own. + Locked: parallel batch runs must share it. """ + import fcntl # Unix only, like the local sandboxes + with _runs_root_lock: if not _runs_root: - root = os.path.realpath(tempfile.mkdtemp(prefix='deepgym_runs_')) + _remove_stale_roots() + root = os.path.realpath(tempfile.mkdtemp(prefix=_RUNS_PREFIX)) + # Locked before it takes the name another process looks for, so it is never + # seen unlocked while this process lives. The kernel drops the lock however + # this process ends; the descriptor stays open until then. + fd = os.open(f'{root}/.locking', os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600) + fcntl.flock(fd, fcntl.LOCK_EX) + os.rename(f'{root}/.locking', f'{root}/{_RUNS_LOCK}') atexit.register(shutil.rmtree, root, ignore_errors=True) _runs_root.append(root) return _runs_root[0] +def _remove_stale_roots() -> None: + """Remove this user's runs roots whose process is gone: killed, it never ran its exit + cleanup. A root whose lock another process holds, or that has none, is left alone.""" + import fcntl + + for root in Path(tempfile.gettempdir()).glob(f'{_RUNS_PREFIX}*'): + try: + if root.is_symlink() or root.stat().st_uid != os.getuid(): + continue + fd = os.open(root / _RUNS_LOCK, os.O_RDONLY | os.O_NOFOLLOW) + except OSError: + continue + try: + fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError: + continue # its process is alive + finally: + os.close(fd) + shutil.rmtree(root, ignore_errors=True) + + +def _deepgym_packages() -> list[str]: + """Every deepgym package a solution's Python could import: this one, and any other + copy on the import path (an installed wheel next to a source checkout, say).""" + here = os.path.realpath(Path(__file__).resolve().parent) + found = (os.path.realpath(Path(entry or os.curdir, 'deepgym')) for entry in sys.path) + others = [path for path in found if Path(path, '__init__.py').is_file()] + return list(dict.fromkeys([here, *others])) + + +def sandbox_python() -> str: + """This Python, by a path a local sandbox has (see :func:`local_isolation`). + + The sandbox holds the installation's real directories, so an alias outside them (a + ``~/bin`` link, a profile link) goes by its target. A venv keeps its own directory, by + its real path, so the venv and its packages stay in use. + """ + executable = os.path.abspath(sys.executable) + folder = os.path.realpath(os.path.dirname(executable)) + prefixes = {os.path.realpath(p) for p in (sys.prefix, sys.exec_prefix)} + if any(Path(folder).is_relative_to(prefix) for prefix in prefixes): + return os.path.join(folder, os.path.basename(executable)) + return os.path.realpath(executable) + + def local_isolation(expose: list[str]) -> dict: - """Isolation spec for a local run. + """Isolation spec for a local run: what its solution may read. + + Only the operating system and the software trees a Python links against (see + :data:`_SYSTEM_READ`), the Python installation and this run's own *expose* + directories. Every deepgym package stays unreadable even inside those. - Hides the home directory (repo clones, other worktrees, ~/.deepgym), all - runs, the deepgym package, and /run and /var/run: daemon sockets there (Docker, - D-Bus, systemd) would act outside the sandbox on request, read-only mount or not. - Re-exposes the Python installation and this run's own *expose* directories; the - package is hidden again even if it sits inside the Python installation. + Raises: + SandboxError: If this machine has no working sandbox-exec or bwrap, unless + :data:`ALLOW_UNSANDBOXED_ENV` is set. """ kind = local_isolation_kind() if kind == 'none': - return {'kind': 'none'} - package = os.path.realpath(Path(__file__).resolve().parent) - hide = [str(Path.home()), local_runs_root(), '/run', '/var/run'] + if unsandboxed_allowed(): + return {'kind': 'none'} + raise SandboxError( + 'No working sandbox-exec or bwrap on this machine, so a local solution could ' + 'read every file you can, its verifier included. Install bubblewrap (Linux), ' + f'use a sandbox provider (mode="docker", "daytona", ...), or set ' + f'{ALLOW_UNSANDBOXED_ENV}=1 to run it unsandboxed anyway.' + ) + system = _SYSTEM_READ['darwin' if kind == 'macos' else 'linux'] pythons = {os.path.realpath(p) for p in (sys.prefix, sys.base_prefix, sys.exec_prefix)} + hidden = [path for path in _SYSTEM_HIDE if os.path.exists(path)] return { 'kind': kind, - 'hide': list(dict.fromkeys(os.path.realpath(p) for p in hide if os.path.exists(p))), - 'expose': [*sorted(pythons), *expose], - 'rehide': [package], + # Unresolved: bwrap recreates a symlinked tree (/bin -> usr/bin) as the link. + 'read': [path for path in system if os.path.lexists(path)] + sorted(pythons) + expose, + 'hide': [*_deepgym_packages(), *hidden], } @@ -269,20 +383,21 @@ def local_judge( judge = [sys.executable, *job.verifier_args(judge_dir, solution_path)] deadline = {'DEEPGYM_DEADLINE': str(time.time() + job.timeout), 'TMPDIR': tmp_dir} if confined: - outer = local_isolation([run_dir]) - if outer['kind'] == 'none': + if local_isolation_kind() == 'none': # never unsandboxed, opt-in or not raise SandboxError('confining the judge needs sandbox-exec or bwrap') + outer = local_isolation([run_dir]) + python = sandbox_python() env = { 'PATH': os.environ.get('PATH', os.defpath), - **job.runtime_env(outer, runtime_path, sys.executable, solution_dir), + **job.runtime_env(outer, runtime_path, python, solution_dir), **deadline, } confine = [sys.executable, '-I', '-B', runtime_path, '--confine', tmp_dir] - yield [*confine, *judge], env, judge_dir + yield [*confine, python, *judge[1:]], env, judge_dir return isolation = local_isolation([runtime_dir, solution_dir]) # Verifier temp files (and solution scratch dirs) stay inside this run. - runtime_env = job.runtime_env(isolation, runtime_path, sys.executable, solution_dir) + runtime_env = job.runtime_env(isolation, runtime_path, sandbox_python(), solution_dir) env = {**os.environ, **runtime_env, **deadline} yield judge, env, judge_dir finally: diff --git a/deepgym/src/deepgym/models.py b/deepgym/src/deepgym/models.py index b9aafe50..cd900ea3 100644 --- a/deepgym/src/deepgym/models.py +++ b/deepgym/src/deepgym/models.py @@ -62,7 +62,7 @@ def checked_score(value: object) -> float: except (TypeError, ValueError) as exc: raise InvalidScoreError(f'Verifier returned a non-numeric score: {value!r}') from exc if not -SCORE_TOLERANCE <= score <= 1 + SCORE_TOLERANCE: # also False for NaN - logger.warning('Verifier returned invalid score %r; scoring 0', score) + logger.warning('Verifier returned invalid score %r; no verdict', score) raise InvalidScoreError(f'Verifier returned invalid score {score!r}; must be in [0, 1]') return min(1.0, max(0.0, score)) diff --git a/deepgym/src/deepgym/sandbox.py b/deepgym/src/deepgym/sandbox.py index a1b45ebe..6997b388 100644 --- a/deepgym/src/deepgym/sandbox.py +++ b/deepgym/src/deepgym/sandbox.py @@ -206,19 +206,50 @@ def split_wrapped_output(output: str, marker: str) -> tuple[str, str]: return stdout, decoded.decode('utf-8', 'replace').strip() -def parse_wrapped_output(output: str, marker: str, exit_code: int) -> ExecutionResult: - """Parse the verdict from the stdout of a command wrapped by :func:`wrap_stderr`. +def parse_wrapped_output( + output: str, marker: str, exit_code: int, consulted_mark: str | None = None +) -> ExecutionResult: + """Parse the verdict from the stdout of a command wrapped by :func:`wrap_stderr`, as + :func:`judge_verdict` does. Raises: InvalidScoreError: If the score is NaN, infinite, or outside [0, 1]. VerifierError: If stdout has no protocol JSON. """ stdout, stderr = split_wrapped_output(output, marker) - return ExecutionResult( - parse_verifier_output(stdout.strip(), f'exit_code={exit_code} stderr: {stderr!r}'), - stderr=stderr, - exit_code=exit_code, - ) + return judge_verdict(stdout, stderr, exit_code, consulted_mark) + + +def judge_verdict( + stdout: str, stderr: str, exit_code: int, consulted_mark: str | None = None +) -> ExecutionResult: + """Parse the verdict a judge printed on *stdout*. + + A judge that printed none after announcing the solution's start (*consulted_mark* in + its stdout), or that ended abnormally then (killed by a signal, or any exit but the + protocol's 0, 1 and 2), whatever it printed before, was ended through the solution, + the only untrusted code it ran: killed, starved of memory or processes. That is the + solution's judged 0 (``solution_error``), or ending its judge would dodge the score. + Before then, it has no verdict. + + Raises: + InvalidScoreError: If the score is NaN, infinite, or outside [0, 1]. + VerifierError: If stdout has no protocol JSON before the solution started. + """ + consulted = consulted_mark is not None and consulted_mark in stdout + if consulted and exit_code not in (0, 1, 2): + details = f'The judge ended abnormally (exit {exit_code}) after the solution ran' + return _judged_zero(details, 'solution_error') + try: + verdict = parse_verifier_output(stdout.strip(), f'exit_code={exit_code} stderr: {stderr!r}') + except InvalidScoreError: + raise + except VerifierError as exc: + if not consulted: + raise + details = f'The judge ended without a verdict after the solution ran: {exc}' + return _judged_zero(details, 'solution_error') + return ExecutionResult(verdict, stderr=stderr, exit_code=exit_code) def _judged_zero(details: str, error_type: str) -> ExecutionResult: @@ -297,7 +328,7 @@ def run_judge_in_sandbox(sandbox: Sandbox, job: JudgeJob) -> ExecutionResult: # The provider couldn't run the command or report back: an outage, not a verdict. raise SandboxError(f'Verifier execution failed: {exc}') from exc # Exit codes 0 (passed) and 1 (failed) are normal verifier outcomes, so parse regardless. - return parse_wrapped_output(response.output, marker, response.exit_code) + return parse_wrapped_output(response.output, marker, response.exit_code, job.consulted_mark) # --------------------------------------------------------------------------- @@ -394,12 +425,7 @@ def run(self, job: JudgeJob) -> ExecutionResult: if job.consulted_mark in _printed(exc.stdout): return solution_timeout(job.timeout) raise TimeoutError(f'Local verifier exceeded {job.timeout}s') from exc - stderr = stderr.strip() - return ExecutionResult( - parse_verifier_output(stdout.strip(), f'stderr: {stderr!r}'), - stderr=stderr, - exit_code=proc.returncode, - ) + return judge_verdict(stdout, stderr.strip(), proc.returncode, job.consulted_mark) class AsyncLocalExecutor: @@ -451,11 +477,9 @@ async def read(stream: asyncio.StreamReader, into: bytearray) -> None: if job.consulted_mark in _printed(bytes(stdout)): return solution_timeout(job.timeout) raise TimeoutError(f'Async local verifier exceeded {job.timeout}s') from exc - stderr_text = stderr.decode('utf-8', errors='replace').strip() - return ExecutionResult( - parse_verifier_output( - stdout.decode('utf-8', errors='replace').strip(), f'stderr: {stderr_text!r}' - ), - stderr=stderr_text, - exit_code=proc.returncode if proc.returncode is not None else -1, + return judge_verdict( + stdout.decode('utf-8', errors='replace'), + stderr.decode('utf-8', errors='replace').strip(), + proc.returncode if proc.returncode is not None else -1, + job.consulted_mark, ) diff --git a/deepgym/src/deepgym/sandboxes/cli.py b/deepgym/src/deepgym/sandboxes/cli.py index 8eb619f9..a2192820 100644 --- a/deepgym/src/deepgym/sandboxes/cli.py +++ b/deepgym/src/deepgym/sandboxes/cli.py @@ -21,7 +21,7 @@ from pydantic import BaseModel, ConfigDict -from deepgym.exceptions import SandboxDiedError, SandboxError +from deepgym.exceptions import SandboxDiedError, SandboxError, SandboxUnreachableError from deepgym.sandboxes.protocol import ( TIMEOUT_EXIT_CODES, TRUSTED_PATH, @@ -131,9 +131,16 @@ def exec( ) -> ExecResult: """Run *command* with sh in the container; output is stdout followed by stderr. + A CLI that gives no answer, or a failed exec, is the daemon's failure only if the + daemon can't list containers either: anything else, even an exec that couldn't + start, may come from what ran in the container before (processes or memory it + used up), so it stays the command's outcome. + Raises: SandboxTimeoutError: If the command outruns *timeout* seconds. SandboxDiedError: If the container stopped during the command. + SandboxUnreachableError: If the CLI gave no answer or the exec failed, and the + daemon can't list containers. """ argv = ['exec'] if cwd: @@ -148,10 +155,10 @@ def exec( try: done = self._cli(*argv, timeout=(timeout or 3600) + 15) except subprocess.TimeoutExpired as exc: - printed = (exc.stdout or b'') + (exc.stderr or b'') - raise SandboxTimeoutError( - f'command exceeded {timeout}s', printed.decode('utf-8', 'replace') - ) from exc + printed = ((exc.stdout or b'') + (exc.stderr or b'')).decode('utf-8', 'replace') + if self._gone(): + raise SandboxDiedError(f'Container {self.id} stopped', printed) from exc + raise SandboxTimeoutError(f'command exceeded {timeout}s', printed) from exc output = done.stdout.decode('utf-8', 'replace') + done.stderr.decode('utf-8', 'replace') if timeout and done.returncode in TIMEOUT_EXIT_CODES: if time.monotonic() - start >= timeout: @@ -162,11 +169,16 @@ def exec( return ExecResult(done.returncode, output) def _gone(self) -> bool: - """Whether the container no longer runs; False if the CLI can't list containers.""" + """Whether the container no longer runs. + + Raises: + SandboxUnreachableError: If the CLI can't list containers, so what it reported + about the command can't be trusted either. + """ try: return all(box.id != self.id for box in self.running()) - except SandboxError: - return False + except SandboxError as exc: + raise SandboxUnreachableError(f'{self.cli} cannot list containers: {exc}') from exc def booted_image(self) -> SandboxImage: """Return the image the container runs and its digest, as the CLI reports them. @@ -213,6 +225,8 @@ def upload(self, content: bytes, path: str) -> None: input=content, timeout=120, ) # fmt: skip except subprocess.TimeoutExpired as exc: + if self._gone(): + raise SandboxDiedError(f'Container {self.id} stopped uploading {path}') from exc raise SandboxError(f'Upload of {path} timed out') from exc if done.returncode != 0: message = f'Failed to upload {path}: {done.stderr.decode()[-500:]}' diff --git a/deepgym/src/deepgym/session.py b/deepgym/src/deepgym/session.py index ac2fa0e0..942237cb 100644 --- a/deepgym/src/deepgym/session.py +++ b/deepgym/src/deepgym/session.py @@ -27,7 +27,7 @@ import time from collections.abc import Callable, Mapping, Sequence from pathlib import Path, PurePosixPath -from typing import TYPE_CHECKING, NamedTuple, Protocol +from typing import TYPE_CHECKING, Literal, NamedTuple, Protocol from uuid import uuid4 from pydantic import JsonValue @@ -51,6 +51,7 @@ SOLVER_USER, ids_are_exclusive, local_isolation, + sandbox_python, ) from deepgym.models import ( Action, @@ -152,6 +153,16 @@ """ +class VerifierUploadError(SandboxError): + """Raise when the provider fails to take a session verifier's code. + + It goes to the verifier's own folder, but on the disk the agent shares, so once the + agent has acted it may be why (it filled the disk or its inodes): as for any write + after an action, that is its judged 0 unless the network or provider failed (see + :func:`failure_result`). Before any action, it is ours: no verdict. + """ + + class StepResult(NamedTuple): """Outcome of one agent action.""" @@ -304,12 +315,14 @@ def write_file(self, path: str, content: bytes) -> None: def verify(self, code: str, timeout: int, env: dict[str, str] | None = None) -> VerifierOutput: """Run verifier *code* inline; it shares the agent's user here. - Code it loads through ``deepgym_solution`` runs in the local judge's sandbox. + Code it loads through ``deepgym_solution`` runs in the local judge's sandbox, which + reads the runtime and the agent's working directory: the files the agent wrote. """ + isolation = local_isolation([str(self._runtime.parent), str(self._dir)]) runtime = { - 'DEEPGYM_ISOLATION': json.dumps(local_isolation([str(self._runtime.parent)])), + 'DEEPGYM_ISOLATION': json.dumps(isolation), 'DEEPGYM_RUNTIME': str(self._runtime), - 'DEEPGYM_PYTHON': sys.executable, + 'DEEPGYM_PYTHON': sandbox_python(), 'DEEPGYM_DEADLINE': str(time.time() + timeout), 'DEEPGYM_POLICY_DIR': str(self._dir), } @@ -791,8 +804,7 @@ def verify(self, code: str, timeout: int, env: dict[str, str] | None = None) -> Raises: TimeoutError: If the verifier exceeds *timeout* seconds. SandboxError: If the sandbox stopped (see :class:`~deepgym.sandboxes.Sandbox`). - VerifierError: If the provider fails to take the verifier's code, which is ours - alone. A failed launch isn't: it runs in the agent's work directory. + VerifierUploadError: If the provider fails to take the verifier's code. """ script = f'{_VERIFIER_DIR}/{uuid4().hex}.py' command, marker = wrap_stderr( @@ -805,8 +817,7 @@ def verify(self, code: str, timeout: int, env: dict[str, str] | None = None) -> except (SandboxDiedError, SandboxExpiredError): raise except Exception as exc: - # Our code to our own folder: the provider refusing it is never the agent's. - raise VerifierError(f'Could not upload the verifier: {exc}') from exc + raise VerifierUploadError(f'Could not upload the verifier: {exc}') from exc done = self._sandbox.exec( trusted( f'{deadline}; {command}; status=$?; rm -f {script}; exit $status', @@ -1249,6 +1260,28 @@ def finalize_cli(self) -> None: self._obs = self._obs.model_copy(update={'done': True}) self._trajectory.final_observation = self._obs + def abandon(self, error_type: Literal['provider_error', 'setup_error'], detail: str) -> None: + """End the episode with no verdict: it stopped through nothing the policy did. + + That is the model provider failing (``provider_error``) or the agent harness never + starting the model (``setup_error``). :meth:`score` then returns that result + without running the verifier: whatever state the policy reached, it didn't finish. + A step that failed first keeps its own result. + """ + with self._lock: + if self._failed_result is None: + self._failed_result = RunResult( + score=0.0, + passed=False, + output=f'Not graded: {detail}', + stderr='', + exit_code=1, + execution_time_ms=0.0, + sandbox_id=self.sandbox_id, + error_type=error_type, + ) + self._done = True + def score(self) -> RunResult: """Run the final verifier against the current state. @@ -1290,11 +1323,16 @@ def score(self) -> RunResult: start = time.perf_counter() acted = bool(self._trajectory.steps) try: - (exit_code, stdout, stderr), consulted = self._verify(self._env.final_verifier_code) + code = self._env.final_verifier_code + (exit_code, stdout, stderr), consulted = self._verify(code, acted=acted) except SandboxDiedError as exc: if not acted: return error_result(exc, self.sandbox_id) return failure_result(exc, self.sandbox_id) + except VerifierUploadError as exc: + if not acted: + raise VerifierError(str(exc)) from exc + return failure_result(exc, self.sandbox_id) except Exception as exc: if acted and self._ran_out_the_clock(exc): return failure_result(exc, self.sandbox_id, exhausted=True) @@ -1564,7 +1602,9 @@ def _step_score(self, exit_code: int, stdout: str) -> float: self._step_verdict = {**data, 'exit_code': exit_code} return score - def _verify(self, code: str, env: dict[str, str] | None = None) -> tuple[VerifierOutput, bool]: + def _verify( + self, code: str, env: dict[str, str] | None = None, *, acted: bool = True + ) -> tuple[VerifierOutput, bool]: """Run session verifier *code* against the sandbox as it stands. It gets a fresh ``DEEPGYM_CONSULTED`` mark in its code (see :func:`_guarded`). It @@ -1577,13 +1617,16 @@ def _verify(self, code: str, env: dict[str, str] | None = None) -> tuple[Verifie run, see :func:`~deepgym.sandbox.solution_timeout`), and so is the verifier being killed (``solution_error``). So is a timeout whose output the backend lost (no start line), since nothing then shows the verifier hadn't read the agent's output. A judged - 0 comes back as if the verifier had printed it. + 0 comes back as if the verifier had printed it. All of that needs an agent that has + *acted*: before any action, its work directory holds only what the environment put + there, so none of these failures is its doing (a :class:`VerifierError`). Returns: What the verifier printed, and whether it read the agent's output. Raises: - VerifierError: If the verifier timed out before the mark. + VerifierError: If the verifier timed out before the mark, or failed in any of + these ways before the agent acted. """ mark = uuid4().hex consulted, started = f'\n{mark}\n', f'\n{mark}:start\n' @@ -1591,6 +1634,10 @@ def _verify(self, code: str, env: dict[str, str] | None = None) -> tuple[Verifie try: exit_code, stdout, stderr = self._backend.verify(_guarded(code, mark), timeout, env) except SandboxTimeoutError as exc: + if not acted: + raise VerifierError( + f'The verifier timed out before the agent acted: {exc}' + ) from exc if consulted in exc.output: return _reported(solution_timeout(timeout)), True if started not in exc.output: @@ -1599,7 +1646,7 @@ def _verify(self, code: str, env: dict[str, str] | None = None) -> tuple[Verifie raise VerifierError( f"The verifier timed out before it read the agent's output: {exc}" ) from exc - read = consulted in stdout + read = acted and consulted in stdout if read and not 0 <= exit_code < 128: details = f"The verifier was killed (exit {exit_code}) handling the agent's output" return _reported(_judged_zero(details, 'solution_error')), True @@ -1664,7 +1711,7 @@ def failure_result(exc: BaseException, sandbox_id: str, *, exhausted: bool = Fal * a network or provider failure, or the provider refusing our credentials (see :func:`~deepgym.exceptions.is_infrastructure_failure`); - * a verifier that couldn't run (its upload or launch failed), failed, or timed out + * a verifier that couldn't run (its launch failed), failed, or timed out before it read the agent's output (:class:`~deepgym.exceptions.VerifierError`; after, the verifier reports the agent's 0 itself, see :meth:`Session._verify`); * the provider retiring the sandbox at the end of its lifetime diff --git a/deepgym/src/deepgym/solution_runtime.py b/deepgym/src/deepgym/solution_runtime.py index 39613fcf..8176d5e7 100644 --- a/deepgym/src/deepgym/solution_runtime.py +++ b/deepgym/src/deepgym/solution_runtime.py @@ -57,6 +57,7 @@ import base64 import builtins import contextlib +import errno import functools import importlib.util import json @@ -81,6 +82,8 @@ if hasattr(sys, 'set_int_max_str_digits'): sys.set_int_max_str_digits(0) _MAX_DEPTH = 100 +# The most the judge holds in memory of one protocol message, a script's stdout or stderr, +# or an output file: more is the solution's failure, never cut to a prefix and graded. _MAX_MESSAGE = 64 * 1024 * 1024 _LOAD_TIMEOUT_S = 30.0 _READY = '{"ready": true}\n' @@ -227,30 +230,41 @@ def paths(items: Sequence[str]) -> str: '(literal "/nonexistent")' ) - hide, expose, rehide = spec['hide'], spec.get('expose', []), spec.get('rehide', []) loopback = ( f'(allow network-outbound (remote ip "localhost:{int(loopback_port)}"))' if loopback_port is not None else '' ) - # Later rules win: hide whole trees, then re-expose this run's own directories. - # Hiding denies reading contents and listings, not stat, so tools that resolve - # their working path (git) still work. No Mach IPC or AppleEvents: they let a - # process ask launchd or another app to start something outside the sandbox. + # Later rules win: nothing is readable, then what the spec reads and the run's own + # directories are (and the root directory itself, which dyld lists to start any + # process), then the spec's hidden paths aren't. Denying data, not metadata, so tools + # that resolve their working path (git) still work. No Mach IPC or AppleEvents: they + # let a process ask launchd or another app to start something outside the sandbox. No + # signals beyond the sandbox: the judge and every other process of this user are + # outside it. return ( f'(version 1)(allow default)(deny network*){loopback}(deny mach-lookup)' - '(deny appleevent-send)' - f'(deny file-read-data {paths(hide)})' - f'(allow file-read-data {paths([*expose, *writable])})' - f'(deny file-read-data {paths(rehide)})' + '(deny appleevent-send)(deny signal)(allow signal (target same-sandbox))' + '(deny file-read-data (subpath "/"))' + f'(allow file-read-data (literal "/") {paths([*spec["read"], *writable])})' + f'(deny file-read-data {paths(spec["hide"])})' f'(deny file-write*)(allow file-write* {paths(writable)} (literal "/dev/null"))' - # Without a separate user there is no reliable way to kill a child that - # calls setsid(), so no forking unless the caller needs subprocesses. + # Without a separate user nothing contains a child that calls setsid(), so the + # solution never forks (see run_solution); only a confined judge may. + ('' if allow_fork else '(deny process-fork)') ) +@functools.cache +def _import_blocker() -> str: + """A module that refuses to import, bound over each hidden package's ``__init__.py``.""" + path = os.path.join(os.path.dirname(_runtime()), '_hidden_package.py') + with open(path, 'w', encoding='utf-8') as handle: + handle.write("raise ImportError('this package is hidden from the solution')\n") + return path + + def _share_with_solution(workdir: str) -> None: """Let the solution user write in *workdir* and read what the judge put there. @@ -285,7 +299,8 @@ def _launch_argv( allow_fork: bool = False, loopback_port: int | None = None, ) -> list[str]: - """Wrap *command* to run as the isolated solution: empty env, writes only in *workdir*. + """Wrap *command* to run as the isolated solution: empty env, writes only in *workdir*, + reads only what the spec allows (see :func:`deepgym.judge.local_isolation`). *loopback_port* only matters on macOS, where the solution shares the judge's network stack; elsewhere :func:`_loopback` forwards the port. A ``none`` spec with @@ -305,16 +320,21 @@ def _launch_argv( profile = _macos_profile(spec, writable, allow_fork, loopback_port) return ['sandbox-exec', '-p', profile, *clean, *command] if kind == 'bwrap': - argv = ['bwrap', '--ro-bind', '/', '/', '--dev', '/dev', '--proc', '/proc'] + # A fresh root holding only what the spec reads; the rest of the host isn't there. + argv = ['bwrap'] + for path in spec['read']: + link = os.path.islink(path) + argv += ['--symlink', os.readlink(path), path] if link else ['--ro-bind', path, path] + argv += ['--dev', '/dev', '--proc', '/proc'] for path in spec['hide']: argv += ['--tmpfs', path] - for path in spec.get('expose', []): - argv += ['--ro-bind', path, path] - for path in spec.get('rehide', []): - argv += ['--tmpfs', path] + # An empty directory would still import, as a namespace package. + if os.path.isfile(init := os.path.join(path, '__init__.py')): + argv += ['--ro-bind', _import_blocker(), init] for path in writable: argv += ['--bind', path, path] - argv += ['--unshare-all', '--die-with-parent', '--new-session', '--chdir', workdir] + argv += ['--remount-ro', '/', '--unshare-all', '--die-with-parent', '--new-session'] + argv += ['--chdir', workdir] return [*argv, *clean, *command] if spec.get('no_fork'): return [*clean, _python(), '-I', '-S', '-c', _NO_FORK, *command] @@ -643,19 +663,26 @@ def run_solution( *cwd* (default: a fresh scratch directory) is the only place it may write; pass any output location inside it and read results back with :func:`read_output`. *env* adds variables to an otherwise empty environment. - Anything the script leaves running is killed when it exits. On macOS the - script can't start subprocesses unless *allow_subprocess* is set (a child - that detaches could outlive it there). *loopback_port* lets it connect to a - server the judge runs on 127.0.0.1 at that port; the rest of the network stays - unreachable. + Anything the script leaves running is killed when it exits. On macOS the script + can't start subprocesses, and *allow_subprocess* is refused: no local sandbox there + contains a child that detaches, which could outlive the script and change what the + judge reads next. *loopback_port* lets it connect to a server the judge runs on + 127.0.0.1 at that port; the rest of the network stays unreachable. Raises: subprocess.TimeoutExpired: If it runs past *timeout* (default: the case timeout); a :class:`SolutionError` too. LaunchError: If the process can't be started, or never got Python going, before - any solution code has run; after that, a :class:`SolutionError`. + any solution code has run; after that, a :class:`SolutionError`. Also for + *allow_subprocess* in the macOS sandbox, whatever has run. """ global _consulted + if allow_subprocess and _isolation()['kind'] == 'macos': + raise LaunchError( + 'allow_subprocess needs a sandbox that contains every process the solution ' + 'starts, which sandbox-exec cannot: run this environment on Linux (bwrap) or ' + 'in a sandbox provider (docker, apple, daytona, modal, e2b)' + ) path = os.path.abspath(solution_path) command = [_python(), '-B', path, *args] limit = _call_timeout() if timeout is None else min(timeout, _remaining()) @@ -708,11 +735,22 @@ def run_solution( raise _launch_failure('the solution process did not start') out.seek(0) err.seek(0) - stdout = out.read(_MAX_MESSAGE).decode('utf-8', 'replace') - stderr = err.read(_MAX_MESSAGE).decode('utf-8', 'replace') + stdout, stderr = _captured(out, 'stdout'), _captured(err, 'stderr') return subprocess.CompletedProcess(command, proc.returncode, stdout, stderr) +def _captured(stream: Any, name: str) -> str: + """What a script printed to *name*, whole. + + Raises: + SolutionError: If it printed more than :data:`_MAX_MESSAGE` bytes there. + """ + data = stream.read(_MAX_MESSAGE + 1) + if len(data) > _MAX_MESSAGE: + raise SolutionError(f'the solution printed more than {_MAX_MESSAGE} bytes to {name}') + return data.decode('utf-8', 'replace') + + def _open_inside(path: str, root: str) -> int: """Open *path* for reading, walking from *root* one component at a time. @@ -748,9 +786,17 @@ def read_output(path: str, root: str) -> str: def read_output_bytes(path: str, root: str, limit: int = _MAX_MESSAGE) -> bytes: - """Binary twin of :func:`read_output`, reading at most *limit* bytes.""" + """Binary twin of :func:`read_output`, for a file of at most *limit* bytes. + + Raises: + OSError: As :func:`read_output`, or (EFBIG) if the file holds more than *limit* + bytes: never just a prefix of it. + """ with os.fdopen(_open_inside(path, root), 'rb') as handle: - return handle.read(limit) + data = handle.read(limit + 1) + if len(data) > limit: + raise OSError(errno.EFBIG, f'{path} holds more than {limit} bytes') + return data def solution_source(solution_path: str, name: str | None = None) -> str: diff --git a/deepgym/src/deepgym/traces/capture.py b/deepgym/src/deepgym/traces/capture.py index b4c1bac3..2cfe65c1 100644 --- a/deepgym/src/deepgym/traces/capture.py +++ b/deepgym/src/deepgym/traces/capture.py @@ -19,7 +19,7 @@ from collections.abc import Callable, Sequence from datetime import datetime, timezone from pathlib import Path -from typing import TextIO, TypeVar +from typing import Literal, TextIO, TypeVar from uuid import uuid4 from deepgym.cli_runtime import CliExecution, CliExecutionError, CliRequest, EventObserver @@ -195,6 +195,12 @@ def run_cli(self, request: CliRequest, on_event: EventObserver) -> CliExecution: lambda: self._session.run_cli(request, on_event), ) + def abandon(self, error_type: Literal['provider_error', 'setup_error'], detail: str) -> None: + """End the episode with no verdict, as :meth:`Session.abandon`; :meth:`score` records it.""" + with self._lock: + self._check_open() + self._session.abandon(error_type, detail) + def finalize_cli(self) -> None: """Finalize CLI containment without inventing an action in the trace. diff --git a/deepgym/tests/agents/test_api.py b/deepgym/tests/agents/test_api.py index b4bdd52d..d472717f 100644 --- a/deepgym/tests/agents/test_api.py +++ b/deepgym/tests/agents/test_api.py @@ -330,8 +330,11 @@ def handler(request): session, initial = started(app) with session: result = AnthropicToolAgent('test').run(session, initial, task='T', budget=UNLIMITED) + grade = session.score() - assert result.termination_reason == 'error' + # The provider's stream was cut: nothing the policy did, so no verdict, never a 0. + assert result.termination_reason == 'provider_error' + assert not grade.has_verdict and grade.error_type == 'provider_error' assert not result.trajectory.steps and app.requests == [] # the received call never ran assert result.usage.input_tokens == 100 # the partial usage is kept assert result.cost.usd is None and 'billing unresolved' in result.cost.provenance @@ -374,7 +377,8 @@ def test_anthropic_tool_calls_cut_short_never_run(app, monkeypatch, finished): with session: result = AnthropicToolAgent('test').run(session, initial, task='T', budget=UNLIMITED) - assert result.termination_reason == 'error' + # Stopped at the model's own output limit: the policy's, scored. Cut: the provider's. + assert result.termination_reason == ('error' if finished else 'provider_error') assert app.requests == [] and not result.trajectory.steps # the cut call never ran payload = json.loads(result.messages[-1].content) assert payload['input_json'] == {'0': raw} # every fragment received, kept @@ -385,6 +389,21 @@ def test_anthropic_tool_calls_cut_short_never_run(app, monkeypatch, finished): assert payload['status'] == 'incomplete' +def test_anthropic_overloaded_mid_stream_is_no_verdict(app, monkeypatch): + """A provider error event arrives under the stream's 200; it is still the provider's.""" + error = {'type': 'error', 'error': {'type': 'overloaded_error', 'message': 'Overloaded'}} + body = f'event: message_start\ndata: {json.dumps(START)}\n\n' + body += f'event: error\ndata: {json.dumps(error)}\n\n' + anthropic(monkeypatch, body.encode()) + session, initial = started(app) + with session: + result = AnthropicToolAgent('test').run(session, initial, task='T', budget=UNLIMITED) + grade = session.score() + + assert result.termination_reason == 'provider_error' + assert not grade.has_verdict and grade.error_type == 'provider_error' + + def test_anthropic_delta_the_sdk_cannot_parse_is_kept_and_never_runs(app, monkeypatch): first = '{"method": "POST", "path": "/rest/bug/1/comment", ' events = [ @@ -555,3 +574,25 @@ def test_environment_without_tools_is_refused(agent): session = Session(MultiTurnEnvironment(task='T'), LocalBackend()) with session, pytest.raises(ValueError, match='declares tools'): agent('test').run(session, session.start(), task='T', budget=UNLIMITED) + + +@pytest.mark.parametrize('usage', [None, 'counted']) +def test_openai_failed_response_is_no_verdict_with_or_without_usage(app, monkeypatch, usage): + """A failed response may carry no usage; it is still the provider's failure, never a 0.""" + body = response( + function_call('tracker', GET_BUG), + status='failed', + error={'code': 'server_error', 'message': 'The server had an error'}, + ) + if usage is None: + body['usage'] = None + openai(monkeypatch, body) + session, initial = started(app) + with session: + result = OpenAIToolAgent('test').run(session, initial, task='T', budget=UNLIMITED) + grade = session.score() + + assert result.termination_reason == 'provider_error' + assert not grade.has_verdict and grade.error_type == 'provider_error' + assert app.requests == [] and not result.trajectory.steps # the failed call never ran + assert 'billing unresolved' in result.cost.provenance diff --git a/deepgym/tests/agents/test_cli.py b/deepgym/tests/agents/test_cli.py index ed73f7de..7bf165a8 100644 --- a/deepgym/tests/agents/test_cli.py +++ b/deepgym/tests/agents/test_cli.py @@ -331,6 +331,11 @@ def __init__(self, name: str) -> None: self.trajectory = Trajectory() self.requests: list[CliRequest] = [] self.finalizations = 0 + self.abandoned: list[tuple[str, str]] = [] + + def abandon(self, error_type: str, detail: str) -> None: + self.abandoned.append((error_type, detail)) + self.done = True def finalize_cli(self) -> None: self.finalizations += 1 @@ -799,3 +804,9 @@ def test_preflight_exception_cleans_setup_population( assert not backend.writer_alive assert session.done assert session.trajectory.steps == [] + + +@pytest.mark.parametrize('provider', ['anthropic', 'openai']) +def test_a_cli_stopped_at_the_output_bound_says_so(provider: str) -> None: + """Stopped for printing more than the host keeps: shown as cut short, never an error.""" + assert parser(provider).finish(-9, 'output_limit') == 'output_limit' diff --git a/deepgym/tests/agents/test_cli_compat.py b/deepgym/tests/agents/test_cli_compat.py index 5d5c94ac..19d30bf9 100644 --- a/deepgym/tests/agents/test_cli_compat.py +++ b/deepgym/tests/agents/test_cli_compat.py @@ -10,7 +10,7 @@ import pytest -from deepgym._cli_worker import parse_cli_version +from deepgym._cli_worker import cli_version_supported, parse_cli_version from deepgym.agents.base import Budget, BudgetTracker from deepgym.agents.cli import ClaudeCodeAgent, CodexAgent from deepgym.agents.cli_events import CliEvents @@ -47,9 +47,9 @@ def test_installed_cli_supports_adapter_flags(provider: str, tmp_path: Path) -> timeout=20, env=env, ) - assert parse_cli_version(version.stdout) == release.version, ( + assert cli_version_supported(parse_cli_version(version.stdout), release.version), ( f'Unsupported local {release.command}: {version.stdout.strip()}. ' - f'Install the pinned CLIs with `{cli_install_command()}`, or bump SUPPORTED_CLIS, ' + f'Install the tested CLIs with `{cli_install_command()}`, or bump SUPPORTED_CLIS, ' 'record new stream fixtures, run the compat test, and rebuild images.' ) agent = (ClaudeCodeAgent if provider == 'anthropic' else CodexAgent)(model='compat-only') diff --git a/deepgym/tests/agents/test_cli_versions.py b/deepgym/tests/agents/test_cli_versions.py index e8121956..cc57f4d1 100644 --- a/deepgym/tests/agents/test_cli_versions.py +++ b/deepgym/tests/agents/test_cli_versions.py @@ -150,13 +150,16 @@ def run_cli(self, request, consume): version_error='Unsupported version; bump SUPPORTED_CLIS and rebuild images', ) + session = UnsupportedSession('claude') result = ClaudeCodeAgent(model='m', api_key='fixture', allow_model_credentials=True).run( - cast(Session, UnsupportedSession('claude')), + cast(Session, session), Observation(content='', step=0), task='task', budget=Budget(), ) - assert result.termination_reason == 'error' + # The model never started: nothing the policy did, so the episode gets no verdict. + assert result.termination_reason == 'setup_error' + assert [kind for kind, _ in session.abandoned] == ['setup_error'] assert result.cli_version == '9.9.9' assert 'rebuild images' in result.error assert result.cost.usd == 0 @@ -197,3 +200,20 @@ def run_cli(self, request, consume): budget=Budget(), ) assert result.termination_reason == 'time_budget' + + +@pytest.mark.parametrize( + ('version', 'tested', 'supported'), + [ + ('2.1.302', '2.1.302', True), + ('2.1.309', '2.1.302', True), # a later release in the major line + ('2.4.0', '2.1.302', True), + ('2.1.301', '2.1.302', False), # older than tested + ('3.0.0', '2.1.302', False), # a new major may change flags and events + ('0.162.4', '0.162.1', True), + ('0.163.0', '0.162.1', False), # below 1.0 a minor release may break them + (None, '2.1.302', False), + ], +) +def test_supported_cli_versions(version, tested, supported) -> None: + assert _cli_worker.cli_version_supported(version, tested) is supported diff --git a/deepgym/tests/agents/test_computer.py b/deepgym/tests/agents/test_computer.py index b8afc018..4e5375a2 100644 --- a/deepgym/tests/agents/test_computer.py +++ b/deepgym/tests/agents/test_computer.py @@ -101,6 +101,11 @@ def __init__(self, max_steps: int | None = None): self.done = False self.trajectory = Trajectory() self.observation = Observation(content='', step=0, screenshot='aW1hZ2U=') + self.abandoned: list[tuple[str, str]] = [] + + def abandon(self, error_type: str, detail: str) -> None: + self.abandoned.append((error_type, detail)) + self.done = True def step(self, action: Action, *, timeout=None): assert timeout > 0 @@ -445,10 +450,21 @@ def test_failed_openai_response_cannot_execute_actions(sdk): result = OpenAIComputerAgent('test').run( session, session.observation, task='Task', budget=Budget() ) - assert result.termination_reason == 'error' + assert result.termination_reason == 'provider_error' + assert session.abandoned == [('provider_error', 'Response did not complete: failed')] assert not result.trajectory.steps +def test_failed_openai_response_without_usage_is_still_no_verdict(sdk): + sdk('openai', [openai_response([], status='failed', usage=None)]) + session = FakeSession() + result = OpenAIComputerAgent('test').run( + session, session.observation, task='Task', budget=Budget() + ) + assert result.termination_reason == 'provider_error' + assert [kind for kind, _ in session.abandoned] == ['provider_error'] + + def test_deadline_after_response_prevents_tool_execution(sdk, monkeypatch): clock = [0.0] monkeypatch.setattr('deepgym.agents.base.time.monotonic', lambda: clock[0]) @@ -870,7 +886,8 @@ def client(**kwargs): result = AnthropicComputerAgent('test').run( session, session.observation, task='Task', budget=Budget() ) - assert result.termination_reason == 'error' + assert result.termination_reason == 'provider_error' # the provider cut it: no verdict + assert [kind for kind, _ in session.abandoned] == ['provider_error'] assert not result.trajectory.steps # the half-received click was not executed assert result.usage.input_tokens == ( 0 if failure == 'empty' else 100 diff --git a/deepgym/tests/agents/test_episode.py b/deepgym/tests/agents/test_episode.py index cfd26921..27992ce7 100644 --- a/deepgym/tests/agents/test_episode.py +++ b/deepgym/tests/agents/test_episode.py @@ -64,3 +64,37 @@ def test_legacy_callable_cannot_silently_ignore_budget() -> None: lambda obs: Action(action_type='done'), budget=Budget(), ) + + +class StoppedAgent: + """An agent of the caller's own that builds its AgentResult directly.""" + + def __init__(self, reason: str) -> None: + self.reason = reason + + def run( + self, session: Session, initial: Observation, *, task: str, budget: Budget + ) -> AgentResult: + session.step(Action(action_type='done')) + return AgentResult( + trajectory=session.trajectory, termination_reason=self.reason, error='stopped' + ) + + +@pytest.mark.parametrize('reason', ['provider_error', 'setup_error']) +def test_any_agent_that_stopped_through_no_policy_fault_gets_no_verdict(reason: str) -> None: + """Enforced where every agent's run is scored, not only for agents that use our helper.""" + env = MultiTurnEnvironment(task='finish', final_verifier_code='print(1.0)') + + _, grade = DeepGym(mode='local').run_episode(env, StoppedAgent(reason), budget=Budget()) + + assert not grade.has_verdict and grade.error_type == reason + + +def test_a_run_the_harness_cut_short_is_graded_as_truncated() -> None: + passes = 'import json\nprint(json.dumps({"score": 1.0, "passed": True}))' + env = MultiTurnEnvironment(task='finish', final_verifier_code=passes) + + _, grade = DeepGym(mode='local').run_episode(env, StoppedAgent('output_limit'), budget=Budget()) + + assert grade.has_verdict and grade.truncated diff --git a/deepgym/tests/test_cli.py b/deepgym/tests/test_cli.py index fc164155..b1fa33a6 100644 --- a/deepgym/tests/test_cli.py +++ b/deepgym/tests/test_cli.py @@ -439,6 +439,19 @@ def test_model_in_local_mode_needs_the_flag( assert '--allow-local-exec' in err assert '--mode docker' in err + def test_model_without_a_sandbox_exits_unless_allowed( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture + ) -> None: + monkeypatch.setattr('deepgym.cli.local_isolation_kind', lambda: 'none') + monkeypatch.delenv('DEEPGYM_ALLOW_UNSANDBOXED', raising=False) + args = ['deepgym', 'qa', 'reverse_string', '--model', 'm', '--allow-local-exec'] + + with patch.object(sys, 'argv', args), pytest.raises(SystemExit) as exc: + main() + + assert exc.value.code == 1 + assert 'DEEPGYM_ALLOW_UNSANDBOXED=1' in capsys.readouterr().err + @pytest.mark.parametrize('isolation', ['bwrap', 'none']) def test_model_with_the_flag_proceeds( self, @@ -447,6 +460,8 @@ def test_model_with_the_flag_proceeds( monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture, ) -> None: + monkeypatch.setenv('DEEPGYM_ALLOW_UNSANDBOXED', '1') # needed only where none works + def handler(request: httpx.Request) -> httpx.Response: answer = '```python\ndef reverse_string(s):\n return s[::-1]\n```' return httpx.Response(200, json={'choices': [{'message': {'content': answer}}]}) diff --git a/deepgym/tests/test_cli_runtime.py b/deepgym/tests/test_cli_runtime.py index 11823d00..ef6c8953 100644 --- a/deepgym/tests/test_cli_runtime.py +++ b/deepgym/tests/test_cli_runtime.py @@ -1,7 +1,9 @@ """Offline checks for CLI supervision, retained isolation and root service ordering.""" import base64 +import errno import json +import os import socket from types import SimpleNamespace from urllib.request import proxy_bypass_environment @@ -11,9 +13,9 @@ from deepgym import _cli_worker, cli_runtime from deepgym.cli_runtime import CliExecution, CliRequest -from deepgym.exceptions import ActionError, SandboxError +from deepgym.exceptions import ActionError, SandboxError, VerifierError from deepgym.models import Action, MultiTurnEnvironment -from deepgym.sandboxes import ExecResult +from deepgym.sandboxes import ExecResult, SandboxTimeoutError from deepgym.session import LocalBackend, SandboxBackend, Session from tests.test_sandboxes import _USERS, FakeSandbox @@ -555,3 +557,51 @@ def popen(argv, **kwargs): assert not proxy_bypass_environment(host, bypass), host for variable in ('HTTP_PROXY', 'HTTPS_PROXY', 'http_proxy', 'https_proxy'): assert captured[variable] == 'http://127.0.0.1:12345' + + +@pytest.mark.parametrize('acted', [False, True]) +def test_a_verifier_upload_the_agent_may_have_blocked(monkeypatch, acted: bool) -> None: + """The verifier's folder is on the disk the agent shares: once it acted, a failed upload + (the disk it filled) is its judged 0, like any write; before, no verdict.""" + box = FakeSandbox(False, [_USERS]) + backend = SandboxBackend(box) + + def disk_full(content: bytes, path: str) -> None: + raise OSError(errno.ENOSPC, os.strerror(errno.ENOSPC)) + + session = Session(MultiTurnEnvironment(task='task', final_verifier_code='print(1)'), backend) + session.start() + if acted: + session.step(Action(action_type='bash', content='true')) + monkeypatch.setattr(box, 'upload', disk_full) + + if not acted: + with pytest.raises(VerifierError, match='Could not upload the verifier'): + session.score() + return + result = session.score() + assert (result.has_verdict, result.error_type) == (True, 'policy_sandbox_failure') + + +@pytest.mark.parametrize('acted', [False, True]) +def test_a_verifier_timeout_with_no_output_blames_only_an_agent_that_acted( + monkeypatch, acted: bool +) -> None: + """Before any action the work directory holds only the environment's files: a verifier + timeout whose output was lost is ours then, not a policy zero.""" + backend = SandboxBackend(FakeSandbox(False, [_USERS])) + session = Session(MultiTurnEnvironment(task='task', final_verifier_code='print(1)'), backend) + session.start() + if acted: + session.step(Action(action_type='bash', content='true')) + + def lost(*args, **kwargs): + raise SandboxTimeoutError('verifier exceeded 30s', '') + + monkeypatch.setattr(backend, 'verify', lost) + if not acted: + with pytest.raises(VerifierError, match='before the agent acted'): + session.score() + return + result = session.score() + assert (result.has_verdict, result.error_type) == (True, 'policy_sandbox_failure') diff --git a/deepgym/tests/test_envs_rewards.py b/deepgym/tests/test_envs_rewards.py index bf6d6e02..f6ba537d 100644 --- a/deepgym/tests/test_envs_rewards.py +++ b/deepgym/tests/test_envs_rewards.py @@ -2,8 +2,11 @@ and a failing judge reports a verifier error instead of a score.""" import ast +import os import random import runpy +import shutil +import subprocess import sys import tempfile from pathlib import Path @@ -264,3 +267,96 @@ def test_a_constant_answer_to_a_yes_no_task_earns_nothing( assert result.has_verdict assert result.score == 0.0 + + +def _cli_task_verifier(): + """The cli_task verifier module, its deepgym_solution import served by the runtime.""" + import importlib.util + + sys.modules.setdefault('deepgym_solution', solution_runtime) + spec = importlib.util.spec_from_file_location( + 'cli_task_verifier', ENVS_DIR / 'cli_task' / 'verifier.py' + ) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _git(repo: Path, *args: str) -> None: + env = {**os.environ, 'GIT_CONFIG_GLOBAL': os.devnull, 'GIT_CONFIG_NOSYSTEM': '1'} + subprocess.run(['git', *args], cwd=repo, env=env, check=True, capture_output=True) + + +@pytest.mark.skipif(shutil.which('git') is None, reason='needs git to build the real repo') +def test_cli_task_reads_a_real_commit_of_the_file(tmp_path: Path) -> None: + verifier = _cli_task_verifier() + repo = tmp_path / 'my_repo' + repo.mkdir() + _git(repo, 'init', '-q') + _git(repo, 'checkout', '-q', '-b', 'feature') + (repo / 'hello.txt').write_text('Hello, DeepGym!') + _git(repo, 'add', 'hello.txt') + _git(repo, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'Add hello.txt') + + read = verifier._Repo(str(repo), str(tmp_path)) + + commits = read.commits(read.branches().values()) + assert read.commit(read.head()) is not None + assert [message.strip() for message, _ in commits] == ['Add hello.txt'] + assert read.file_in(commits[0][1], 'hello.txt') == b'Hello, DeepGym!' + + +def test_cli_task_takes_no_forged_git_files(tmp_path: Path) -> None: + """A ref to forty zeroes and an object that merely starts like a commit, with no HEAD or + tree: what used to earn every check, now none of the git ones.""" + import zlib + + verifier = _cli_task_verifier() + git = tmp_path / 'my_repo' / '.git' + (git / 'refs' / 'heads').mkdir(parents=True) + (git / 'refs' / 'heads' / 'feature').write_text('0' * 40) + (git / 'objects' / '00').mkdir(parents=True) + (git / 'objects' / '00' / ('0' * 38)).write_bytes( + zlib.compress(b'commit 999\0\n\nAdd hello.txt') + ) + + read = verifier._Repo(str(tmp_path / 'my_repo'), str(tmp_path)) + + assert read.head() is None + assert read.branches() == {'feature': '0' * 40} + assert read.commit('0' * 40) is None # its content doesn't hash to its name + assert read.commits(read.branches().values()) == [] + + +def _store(git: Path, kind: str, body: bytes) -> str: + """Write a loose object as git does, under its true id; return the id.""" + import hashlib + import zlib + + raw = f'{kind} {len(body)}'.encode() + b'\0' + body + sha = hashlib.sha1(raw).hexdigest() + (git / 'objects' / sha[:2]).mkdir(parents=True, exist_ok=True) + (git / 'objects' / sha[:2] / sha[2:]).write_bytes(zlib.compress(raw)) + return sha + + +def test_cli_task_takes_no_malformed_or_missing_tree(tmp_path: Path) -> None: + """Objects stored under their true ids still prove nothing if git can't parse them: a + tree with a malformed entry after hello.txt, or a commit whose tree doesn't exist.""" + verifier = _cli_task_verifier() + git = tmp_path / 'my_repo' / '.git' + blob = _store(git, 'blob', b'Hello, DeepGym!') + entry = b'100644 hello.txt\0' + bytes.fromhex(blob) + good = _store(git, 'tree', entry) + bad = _store(git, 'tree', entry + b'not a valid tree entry') + who = b'author t 0 +0000\ncommitter t 0 +0000\n\nAdd hello.txt\n' + with_good = _store(git, 'commit', f'tree {good}\n'.encode() + who) + with_bad = _store(git, 'commit', f'tree {bad}\n'.encode() + who) + with_none = _store(git, 'commit', f'tree {"1" * 40}\n'.encode() + who) + + read = verifier._Repo(str(tmp_path / 'my_repo'), str(tmp_path)) + + assert read.file_in(good, 'hello.txt') == b'Hello, DeepGym!' + assert read.commit(with_good) is not None + assert read.entries(bad) is None and read.file_in(bad, 'hello.txt') is None + assert read.commit(with_bad) is None and read.commit(with_none) is None diff --git a/deepgym/tests/test_envs_smoke.py b/deepgym/tests/test_envs_smoke.py index 30b0d4d6..a57430f7 100644 --- a/deepgym/tests/test_envs_smoke.py +++ b/deepgym/tests/test_envs_smoke.py @@ -6,12 +6,21 @@ import pytest from deepgym.core import DeepGym +from deepgym.judge import local_isolation_kind from deepgym.qa import QAOptions, run_qa ENVS_DIR = Path(__file__).resolve().parent.parent / 'src' / 'deepgym' / 'envs' _BUILTIN = json.loads((ENVS_DIR / 'registry.json').read_text(encoding='utf-8'))['environments'] -_SINGLE_TURN_ENVS = [entry['path'] for entry in _BUILTIN if entry.get('type') != 'computer-use'] +# cli_task's verifier runs the solution with subprocesses (git), which macOS refuses. +_NEEDS_SUBPROCESSES = pytest.mark.skipif( + local_isolation_kind() == 'macos', reason='allow_subprocess needs bwrap or a sandbox provider' +) +_SINGLE_TURN_ENVS = [ + pytest.param(entry['path'], marks=_NEEDS_SUBPROCESSES if entry['path'] == 'cli_task' else ()) + for entry in _BUILTIN + if entry.get('type') != 'computer-use' +] @pytest.fixture(scope='module') diff --git a/deepgym/tests/test_exceptions.py b/deepgym/tests/test_exceptions.py index 5e7c515b..39326934 100644 --- a/deepgym/tests/test_exceptions.py +++ b/deepgym/tests/test_exceptions.py @@ -159,3 +159,25 @@ def test_modal_errors_are_told_apart() -> None: assert is_infrastructure_failure(modal.exception.ConnectionError('no route')) assert not is_infrastructure_failure(modal.exception.SandboxTerminatedError('terminated')) assert not is_infrastructure_failure(modal.exception.ExecTimeoutError('slow')) + + +@pytest.mark.parametrize( + ('module', 'name'), + [ + ('e2b.exceptions', 'AuthenticationException'), + ('modal.exception', 'AuthError'), + ('modal.exception', 'PermissionDeniedError'), + ], +) +def test_provider_auth_errors_without_an_http_status_are_infrastructure( + monkeypatch: pytest.MonkeyPatch, module: str, name: str +) -> None: + """Our credentials refused by an SDK whose error carries no integer status (E2B's has + none, Modal's is an Enum): never the policy's zero.""" + import sys + import types + + error = type(name, (Exception,), {}) + monkeypatch.setitem(sys.modules, module, types.SimpleNamespace(**{name: error})) + + assert is_infrastructure_failure(_caused_by(SandboxError('exec failed'), error('denied'))) diff --git a/deepgym/tests/test_import_bigcodebench.py b/deepgym/tests/test_import_bigcodebench.py index 53293b0d..5c9eb31e 100644 --- a/deepgym/tests/test_import_bigcodebench.py +++ b/deepgym/tests/test_import_bigcodebench.py @@ -6,6 +6,7 @@ import pytest from deepgym import DeepGym +from deepgym.judge import local_isolation_kind from deepgym.models import Environment scripts_dir = Path(__file__).resolve().parent.parent / 'scripts' @@ -39,6 +40,12 @@ def test_large(self): _SKIP_NEGATIVE = _SKIP + ' if x < 0:\n raise unittest.SkipTest("no")\n return x * x\n' +# Its verifier runs the task's tests as a script with subprocesses, which macOS refuses. +needs_subprocesses = pytest.mark.skipif( + local_isolation_kind() == 'macos', reason='allow_subprocess needs bwrap or a sandbox provider' +) + + @pytest.fixture(scope='module') def env() -> Environment: """A five-test BigCodeBench-style task: square a number.""" @@ -51,6 +58,7 @@ def dg() -> DeepGym: return DeepGym(mode='local') +@needs_subprocesses @pytest.mark.parametrize( ('solution', 'score', 'passed'), [ @@ -86,6 +94,7 @@ def test_minus_three(self): """ +@needs_subprocesses def test_skipped_subtests_count_once_per_test(dg: DeepGym) -> None: """Five skipped subtests of one method used to give 3 - 5 = -2 passing tests.""" env = Environment(task='t', verifier_code=generate_verifier('t', 'task_func', _SUBTESTS, [])) diff --git a/deepgym/tests/test_isolation.py b/deepgym/tests/test_isolation.py index 7d7aef70..b6f066ef 100644 --- a/deepgym/tests/test_isolation.py +++ b/deepgym/tests/test_isolation.py @@ -112,7 +112,7 @@ def test_a_cleanup_the_solution_broke_is_its_failure( script = tmp_path / 'solution.py' script.write_text('print("ran")\n', encoding='utf-8') - def no_processes_left(proc: subprocess.Popen) -> None: + def no_processes_left(proc: subprocess.Popen, mark: str | None = None) -> None: proc.kill() raise OSError(errno.EAGAIN, 'Resource temporarily unavailable') @@ -123,6 +123,24 @@ def no_processes_left(proc: subprocess.Popen) -> None: assert solution_runtime.consulted() +def test_output_past_the_bound_is_the_solutions_failure_not_a_prefix( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A grader must never see a cut-off prefix of what the solution printed or wrote.""" + from deepgym import solution_runtime + + monkeypatch.setattr(solution_runtime, '_MAX_MESSAGE', 16) + script = tmp_path / 'solution.py' + script.write_text('print("right answer, then more")\n', encoding='utf-8') + with pytest.raises(solution_runtime.SolutionError, match='more than 16 bytes to stdout'): + solution_runtime.run_solution(str(script), cwd=str(tmp_path)) + + (tmp_path / 'out.bin').write_bytes(b'x' * 17) + with pytest.raises(OSError, match='more than 16 bytes'): + solution_runtime.read_output_bytes(str(tmp_path / 'out.bin'), str(tmp_path), limit=16) + assert solution_runtime.read_output_bytes(str(tmp_path / 'out.bin'), str(tmp_path), 17) + + def test_a_script_runs_only_once_the_watcher_knows_it( tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: @@ -250,16 +268,92 @@ def no_threads(self: object) -> None: assert started[-1].poll() is not None # the server it started is gone -def test_local_sandboxes_hide_host_bus_sockets(monkeypatch: pytest.MonkeyPatch) -> None: +@pytest.mark.parametrize('kind', ['bwrap', 'macos']) +def test_local_sandboxes_read_nothing_a_user_or_process_wrote( + kind: str, monkeypatch: pytest.MonkeyPatch +) -> None: + """Only system trees, the Python installation and the run's own directories are + readable: never home, temp directories (other runs) or daemon sockets.""" import deepgym.judge + monkeypatch.setattr(deepgym.judge, 'local_isolation_kind', lambda: kind) + + read = deepgym.judge.local_isolation(['/the/run'])['read'] + + pythons = {os.path.realpath(p) for p in (sys.prefix, sys.base_prefix, sys.exec_prefix)} + places = [Path.home(), Path(tempfile.gettempdir()), Path('/tmp'), Path('/run'), Path('/var')] + for place in (Path(os.path.realpath(p)) for p in places): + covering = [path for path in read if place.is_relative_to(os.path.realpath(path))] + assert not covering, (place, covering) + assert pythons <= set(read) + assert read[-1] == '/the/run' + + +def test_every_deepgym_copy_on_the_path_stays_hidden( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A wheel installed next to a source checkout is as unreadable as the copy running.""" + import deepgym.judge + + copy = tmp_path / 'deepgym' + copy.mkdir() + (copy / '__init__.py').write_text('') + monkeypatch.setattr(sys, 'path', [*sys.path, str(tmp_path)]) monkeypatch.setattr(deepgym.judge, 'local_isolation_kind', lambda: 'bwrap') - monkeypatch.setattr(deepgym.judge.os.path, 'exists', lambda path: True) hide = deepgym.judge.local_isolation([])['hide'] - assert {'/run', os.path.realpath('/var/run')} <= set(hide) - assert len(hide) == len(set(hide)) + assert os.path.realpath(copy) in hide + assert os.path.realpath(Path(deepgym.judge.__file__).parent) in hide + + +def test_no_sandbox_refuses_to_run_unless_allowed(monkeypatch: pytest.MonkeyPatch) -> None: + import deepgym.judge + + monkeypatch.setattr(deepgym.judge, 'local_isolation_kind', lambda: 'none') + monkeypatch.delenv(deepgym.judge.ALLOW_UNSANDBOXED_ENV, raising=False) + with pytest.raises(SandboxError, match='No working sandbox-exec or bwrap'): + DeepGym(mode='local').run(load_environment('coin_change'), WRONG) + + monkeypatch.setenv(deepgym.judge.ALLOW_UNSANDBOXED_ENV, '1') + assert deepgym.judge.local_isolation([]) == {'kind': 'none'} + + +_RUNS_ROOT_THEN = ( + 'import os, signal, sys, time\n' + 'from deepgym.judge import local_runs_root\n' + 'print(local_runs_root(), flush=True)\n' + 'os.kill(os.getpid(), signal.SIGKILL) if sys.argv[1] == "die" else time.sleep(60)\n' +) + + +def test_a_killed_process_runs_root_is_removed_by_the_next( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A killed process never runs its exit cleanup; the next one removes what it left, + and never the root of a process still running.""" + import deepgym.judge + + env = {**os.environ, 'TMPDIR': str(tmp_path)} + live = subprocess.Popen( + [sys.executable, '-c', _RUNS_ROOT_THEN, 'live'], env=env, stdout=subprocess.PIPE, text=True + ) + try: + live_root = Path(live.stdout.readline().strip()) + dead = subprocess.run( + [sys.executable, '-c', _RUNS_ROOT_THEN, 'die'], env=env, capture_output=True, text=True + ) + dead_root = Path(dead.stdout.strip()) + assert dead_root.is_dir() and live_root.is_dir() + monkeypatch.setattr(tempfile, 'tempdir', str(tmp_path)) + + deepgym.judge._remove_stale_roots() + + assert not dead_root.exists() + assert live_root.is_dir() + finally: + live.kill() + live.wait() _LAUNCHER = """ @@ -386,6 +480,33 @@ def test_cannot_read_judge_files_or_import_deepgym(self, dg: DeepGym) -> None: result = dg.run(env, solution) assert result.passed, result.output + @needs_sandbox + def test_cannot_read_temp_files_or_other_runs(self, dg: DeepGym) -> None: + """Nothing outside its run is readable: a file in the temp directory, or another + DeepGym process's runs (another trainer worker's verifier copy).""" + other = Path(tempfile.mkdtemp(prefix='deepgym_runs_')) + loose = Path(tempfile.mkstemp()[1]) + try: + (other / 'verifier.py').write_text('ANSWER = 42') + loose.write_text('answer key') + targets = [str(other / 'verifier.py'), str(loose)] + env = _env( + f'leaks = [path for path in {targets!r} if sol.read(path) is not None]\n' + "return {'score': float(not leaks), 'passed': not leaks, 'details': str(leaks)}" + ) + solution = ( + 'def read(path):\n' + ' try:\n' + ' return open(path).read()\n' + ' except OSError:\n' + ' return None\n' + ) + result = dg.run(env, solution) + assert result.passed, result.output + finally: + shutil.rmtree(other) + loose.unlink() + def test_host_secrets_not_visible(self, dg: DeepGym, monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setenv('DG_TEST_SECRET', 'hunter2') env = _env("return {'score': 0.0 if sol.leaked() else 1.0, 'passed': not sol.leaked()}") @@ -890,16 +1011,41 @@ def test_functions_cannot_fork(self, dg: DeepGym) -> None: assert dg.run(env, solution).passed @macos_only - def test_scripts_fork_only_when_allowed(self, dg: DeepGym) -> None: - verifier = ( + def test_scripts_cannot_fork_and_allow_subprocess_is_refused(self, dg: DeepGym) -> None: + """No macOS sandbox contains a child that detaches, so a script never forks there, + and asking for subprocesses ends the run with no verdict rather than allow it.""" + blocked = ( 'from deepgym_solution import run_solution\n\n' 'def verify(solution_path):\n' - ' blocked = run_solution(solution_path).returncode != 0\n' - ' allowed = run_solution(solution_path, allow_subprocess=True).returncode == 0\n' - ' return blocked and allowed\n' + ' return run_solution(solution_path).returncode != 0\n' + ) + allowed = ( + 'from deepgym_solution import run_solution\n\n' + 'def verify(solution_path):\n' + ' return run_solution(solution_path, allow_subprocess=True).returncode == 0\n' ) solution = 'import subprocess\nsubprocess.run(["true"], check=True)\n' - assert dg.run(Environment(task='t', verifier_code=verifier), solution).passed + assert dg.run(Environment(task='t', verifier_code=blocked), solution).passed + + refused = dg.run(Environment(task='t', verifier_code=allowed), solution) + + assert not refused.has_verdict + assert 'allow_subprocess needs a sandbox' in refused.output + refused.stderr + + @macos_only + def test_cannot_signal_the_judge(self, dg: DeepGym) -> None: + """Signals reach only the sandbox: not the judge, whose death would cost the verdict.""" + env = _env('return {"score": 0.0 if sol.signal_parent() else 1.0, "passed": True}') + solution = ( + 'import os\n' + 'def signal_parent():\n' + ' try:\n' + ' os.kill(os.getppid(), 0)\n' + ' return True\n' + ' except OSError:\n' + ' return False\n' + ) + assert dg.run(env, solution).score == 1.0 LOOPBACK_PROBE_VERIFIER = """ @@ -1249,3 +1395,16 @@ def test_judge_setup_runs_the_solution_only_as_an_account_of_its_own( else: assert setup.returncode != 0 assert 'dgsolver is root or shares an id or a group' in setup.stdout + + +def test_a_python_alias_outside_the_installation_goes_by_its_target( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A ~/bin link to the interpreter isn't in a local sandbox's fresh root; its target is.""" + import deepgym.judge + + alias = tmp_path / 'python3' + alias.symlink_to(os.path.realpath(sys.executable)) + monkeypatch.setattr(sys, 'executable', str(alias)) + + assert deepgym.judge.sandbox_python() == os.path.realpath(sys.executable) diff --git a/deepgym/tests/test_isolation_live.py b/deepgym/tests/test_isolation_live.py index 605bd9a2..b0be98c6 100644 --- a/deepgym/tests/test_isolation_live.py +++ b/deepgym/tests/test_isolation_live.py @@ -64,7 +64,7 @@ def test_solution_user_cannot_touch_judge(dg: DeepGym) -> None: ' "read_secret": lambda: open(secret).read(),\n' ' "list_judge": lambda: os.listdir(os.path.dirname(secret)),\n' ' "judge_stdout": lambda: open(f"/proc/{judge_pid}/fd/1", "w").write("x"),\n' - ' "signal_judge": lambda: os.kill(judge_pid, 0),\n' + ' "signal_judge": lambda: os.kill(judge_pid, 0) is None, # it returns None\n' ' "is_root": lambda: os.getuid() == 0 or None,\n' ' }.items():\n' ' try:\n' diff --git a/deepgym/tests/test_local_executor.py b/deepgym/tests/test_local_executor.py index fff69f58..11cee914 100644 --- a/deepgym/tests/test_local_executor.py +++ b/deepgym/tests/test_local_executor.py @@ -14,7 +14,7 @@ from deepgym.exceptions import DeepGymError, SandboxError, TimeoutError, VerifierError from deepgym.judge import JudgeJob from deepgym.models import Environment, VerifierResult -from deepgym.sandbox import AsyncLocalExecutor, ExecutionResult, LocalExecutor +from deepgym.sandbox import AsyncLocalExecutor, ExecutionResult, LocalExecutor, judge_verdict ENVS_DIR = Path(__file__).resolve().parent.parent / 'src' / 'deepgym' / 'envs' @@ -364,3 +364,44 @@ def test_string_manipulation_example(self, executor: LocalExecutor) -> None: ) assert result.verifier_result.passed is True assert result.verifier_result.score == 1.0 + + +_JUDGE_DIES = """\ +import os, signal +from deepgym_solution import load_solution + +def verify(solution_path, test_cases_path=None): + if os.environ['WHEN'] == 'after': + load_solution(solution_path).f() + os.kill(os.getpid(), signal.SIGKILL) +""" + + +@pytest.mark.parametrize('executor', [LocalExecutor, AsyncLocalExecutor]) +def test_a_judge_that_dies_after_the_solution_ran_is_its_judged_zero(executor) -> None: + """Killed or starved through what the solution ran, the judge printed no verdict: the + solution's 0, or ending the judge would dodge it. Before the solution ran, no verdict.""" + + def run(when: str) -> ExecutionResult: + env = Environment(task='t', verifier_code=_JUDGE_DIES, env_vars={'WHEN': when}) + job = JudgeJob.for_environment(env, 'def f():\n return 1\n') + result = executor().run(job) + return asyncio.run(result) if asyncio.iscoroutine(result) else result + + after = run('after').verifier_result + assert (after.score, after.error_type) == (0.0, 'solution_error') + with pytest.raises(VerifierError): + run('before') + + +def test_a_judge_killed_after_an_earlier_report_is_still_the_solutions_zero() -> None: + """A report printed before a later solution call can't make the judge's death after it + anyone else's: the abnormal exit decides, whatever was printed.""" + stdout = 'mark\n{"score": 1.0, "passed": true}\n' + + killed = judge_verdict(stdout, '', -9, 'mark').verifier_result + + assert (killed.score, killed.error_type) == (0.0, 'solution_error') + assert judge_verdict(stdout, '', 0, 'mark').verifier_result.score == 1.0 + with pytest.raises(VerifierError): + judge_verdict('', '', -9, 'mark') # nothing announced: the solution never ran diff --git a/deepgym/tests/test_qa_mutants.py b/deepgym/tests/test_qa_mutants.py index 69124d93..498d8a63 100644 --- a/deepgym/tests/test_qa_mutants.py +++ b/deepgym/tests/test_qa_mutants.py @@ -2,7 +2,9 @@ import ast import os +import random import signal +import subprocess import time from pathlib import Path @@ -317,21 +319,29 @@ def test_a_confined_run_takes_down_what_the_judge_left_running() -> None: ' exec(open(solution_path).read(), namespace)\n' " return {'score': 1.0, 'passed': True, 'details': str(namespace['start']())}\n" ) - worker = 'import subprocess\n\ndef start():\n return subprocess.Popen(["sleep", "60"]).pid\n' + # Found by its unique command line: under bwrap the pid it reports is its namespace's. + command = f'sleep {60 + random.random():.6f}' + worker = ( + f'import subprocess\n\ndef start():\n return subprocess.Popen({command.split()!r}).pid\n' + ) env = Environment(task='t', verifier_code=verifier) - result = DeepGym(mode='local', confine_local_judge=True).run(env, worker) + DeepGym(mode='local', confine_local_judge=True).run(env, worker) - pid = int(result.output) for _ in range(50): # the kill is sent before run() returns; give the OS a moment - try: - os.kill(pid, 0) - except ProcessLookupError: + if not (left := _pids_running(command)): break time.sleep(0.1) else: - os.kill(pid, signal.SIGKILL) - pytest.fail(f'worker {pid} outlived its confined run') + for pid in left: + os.kill(pid, signal.SIGKILL) + pytest.fail(f'worker {left} outlived its confined run') + + +def _pids_running(command: str) -> list[int]: + listing = subprocess.run(['ps', '-axo', 'pid=,args='], capture_output=True, text=True).stdout + rows = (line.strip().split(None, 1) for line in listing.splitlines()) + return [int(row[0]) for row in rows if len(row) == 2 and row[1] == command] @pytest.mark.skipif(local_isolation_kind() != 'macos', reason='sandbox-exec only') diff --git a/deepgym/tests/test_sandboxes.py b/deepgym/tests/test_sandboxes.py index ef883660..71e0c9a6 100644 --- a/deepgym/tests/test_sandboxes.py +++ b/deepgym/tests/test_sandboxes.py @@ -26,6 +26,7 @@ DeepGymError, SandboxError, SandboxExpiredError, + SandboxUnreachableError, VerifierError, is_infrastructure_failure, ) @@ -51,7 +52,7 @@ from deepgym.sandboxes.cli import DEFAULT_IMAGE from deepgym.sandboxes.daytona import cleanup_sandbox, snapshot_digest from deepgym.sandboxes.docker import DockerContainer -from deepgym.session import _SEALED_PATH_CHECK, SandboxBackend, Session +from deepgym.session import _SEALED_PATH_CHECK, SandboxBackend, Session, VerifierUploadError # Fake sandboxes pass session setup's check that only root can change TRUSTED_PATH. _SEALED_NEEDLE = 'root would run what others can change' @@ -200,6 +201,70 @@ def fake_run(argv: list[str], **kwargs: object) -> subprocess.CompletedProcess: box.exec('sleep 99', timeout=5) +def _exec_answering( + monkeypatch: pytest.MonkeyPatch, exec_answer, listing: Exception | None = None +) -> DockerContainer: + """Fake the docker CLI: exec runs *exec_answer*; the listing shows the container running, + or raises *listing* (a daemon that can't answer).""" + + def fake_run(argv: list[str], **kwargs: object) -> subprocess.CompletedProcess: + if argv[1] == 'exec': + return exec_answer(argv) + return subprocess.CompletedProcess(argv, 0, b'', b'') + + monkeypatch.setattr(subprocess, 'run', fake_run) + box = DockerContainer(image='img') + + def running() -> list[types.SimpleNamespace]: + if listing is not None: + raise listing + return [types.SimpleNamespace(id=box.id)] + + monkeypatch.setattr(box, 'running', running) + return box + + +@pytest.mark.parametrize('daemon_answers', [True, False]) +def test_a_cli_that_gives_no_answer_is_ours_only_if_the_daemon_cannot_answer( + monkeypatch: pytest.MonkeyPatch, daemon_answers: bool +) -> None: + """A daemon that still lists the running container was held by what ran in it (a child + left with the output open, processes or memory used up): the command's own timeout. + One that can't list containers failed itself: infrastructure, no verdict.""" + + def hang(argv: list[str]) -> subprocess.CompletedProcess: + raise subprocess.TimeoutExpired(argv, 20, output=b'partial') + + listing = None if daemon_answers else SandboxError('docker could not list containers') + box = _exec_answering(monkeypatch, hang, listing) + + with pytest.raises( + SandboxTimeoutError if daemon_answers else SandboxUnreachableError + ) as caught: + box.exec('work', timeout=5) + assert is_infrastructure_failure(caught.value) is not daemon_answers + + +@pytest.mark.parametrize('daemon_answers', [True, False]) +def test_a_failed_exec_is_the_commands_unless_the_daemon_cannot_answer( + monkeypatch: pytest.MonkeyPatch, daemon_answers: bool +) -> None: + """Even an exec that couldn't start (the container's process limit, used up by what ran + before) stays the command's outcome while the daemon answers.""" + + def fails(argv: list[str]) -> subprocess.CompletedProcess: + return subprocess.CompletedProcess(argv, 126, b'', b'cannot start: resource limit') + + listing = None if daemon_answers else SandboxError('Cannot connect to the Docker daemon') + box = _exec_answering(monkeypatch, fails, listing) + + if daemon_answers: + assert box.exec('work', timeout=5) == ExecResult(126, 'cannot start: resource limit') + return + with pytest.raises(SandboxUnreachableError): + box.exec('work', timeout=5) + + def test_cli_start_timeout_removes_the_container(monkeypatch: pytest.MonkeyPatch) -> None: calls: list[list[str]] = [] @@ -638,7 +703,7 @@ def upload(content: bytes, path: str) -> None: raise refused box.upload = upload # type: ignore[method-assign] - with pytest.raises(VerifierError, match='Could not upload the verifier'): + with pytest.raises(VerifierUploadError, match='Could not upload the verifier'): session.step(Action(action_type='bash', content='echo hi')) result = session.score() @@ -1000,9 +1065,11 @@ def fake_run(argv: list[str], **kwargs: object) -> subprocess.CompletedProcess: return subprocess.CompletedProcess(argv, 0, b'', b'') monkeypatch.setattr(subprocess, 'run', fake_run) + box = DockerContainer() + monkeypatch.setattr(box, 'running', lambda: [types.SimpleNamespace(id=box.id)]) with pytest.raises(SandboxTimeoutError) as raised: - DockerContainer().exec('sleep 9', timeout=1) + box.exec('sleep 9', timeout=1) assert raised.value.output == 'printed first' @@ -1295,9 +1362,10 @@ def test_the_consulted_mark_is_in_the_private_script_never_on_a_command_line() - assert not any('DEEPGYM_CONSULTED' in command for command in box.commands) -def test_only_uploading_the_verifier_is_ours_whatever_the_provider_says() -> None: - """Its code to its own folder is ours alone. Its launch is not: it runs in the agent's - work directory, which the agent can move away.""" +def test_a_failed_verifier_upload_is_told_apart_from_its_launch() -> None: + """Its code goes to its own folder, on the disk the agent shares: a failed upload is a + VerifierUploadError, whose verdict turns on whether the agent acted (Session.score). + Its launch runs in the agent's work directory, which the agent can move away.""" box = FakeSandbox(network_blocked=True, answers=[_USERS]) backend = SandboxBackend(box) real_upload, real_exec = box.upload, box.exec @@ -1306,7 +1374,7 @@ def upload(content: bytes, path: str) -> None: raise _ProviderError('HTTP 400: bad request') box.upload = upload # type: ignore[method-assign] - with pytest.raises(VerifierError, match='Could not upload the verifier'): + with pytest.raises(VerifierUploadError, match='Could not upload the verifier'): backend.verify('print(1)', timeout=5) def exec_(command: str, **kwargs: object) -> ExecResult: @@ -1318,10 +1386,10 @@ def exec_(command: str, **kwargs: object) -> ExecResult: with pytest.raises(_ProviderError): backend.verify('print(1)', timeout=5) - # Docker and Apple report an ordinary failed upload as a SandboxError: still ours. A - # sandbox found stopped is not. + # Docker and Apple report an ordinary failed upload as a SandboxError. A sandbox found + # stopped stays that. for error, raised in ( - (SandboxError('Failed to upload: no space left'), VerifierError), + (SandboxError('Failed to upload: no space left'), VerifierUploadError), (SandboxDiedError('the container stopped'), SandboxDiedError), ):