fix: sandbox allowlist, no verdict for provider failures

.github/workflows/deepgym-ci.yml

@@ -36,8 +36,16 @@ jobs:
           cache: pip
           cache-dependency-path: deepgym/pyproject.toml
 
+      - name: Install bubblewrap (the local sandbox)
+        run: |
+          sudo apt-get update -q && sudo apt-get install -y -q bubblewrap
+          # Ubuntu 24.04 kernels with AppArmor block the unprivileged user namespaces bwrap needs.
+          knob=/proc/sys/kernel/apparmor_restrict_unprivileged_userns
+          if [ -e "$knob" ]; then sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0; fi
+          bwrap --ro-bind / / --unshare-all true
+
       - name: Install dependencies
-        run: pip install -e ".[dev]"
+        run: pip install -e ".[dev,anthropic,openai]"
 
       - name: Lint (ruff)
         run: ruff check src/ tests/
@@ -64,6 +72,13 @@ jobs:
           cache: pip
           cache-dependency-path: deepgym/pyproject.toml
 
+      - name: Install bubblewrap (the local sandbox)
+        run: |
+          sudo apt-get update -q && sudo apt-get install -y -q bubblewrap
+          knob=/proc/sys/kernel/apparmor_restrict_unprivileged_userns
+          if [ -e "$knob" ]; then sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0; fi
+          bwrap --ro-bind / / --unshare-all true
+
       # CPU-only torch keeps the job small. TRL 1.15 computes GRPO log-probs with Triton GPU
       # kernels, so CPU runners stay on TRL 1.14, whose reward and advantage code is the same.
       - name: Install dependencies

deepgym/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## Unreleased
+
+- **Local sandbox reads only what a run needs:** under `sandbox-exec` and `bwrap`, a solution can read the operating system, the software trees a Python links against, the Python installation and its own run's files, and nothing else. Before, only the home directory, this process's runs and daemon sockets were hidden, so a solution could read files under `/tmp` and `$TMPDIR`, including another DeepGym process's verifier copies. Every deepgym package on the import path stays unreadable, not just the one running. Runs roots that a killed process left behind are removed by the next process.
+- **macOS sandbox:** a solution can't signal anything outside its sandbox, its judge included. Nothing on macOS contains a child that detaches, so `run_solution(..., allow_subprocess=True)` is refused there (no verdict); run such environments on Linux or a sandbox provider.
+- **A judge killed after the solution ran is its 0:** a local or sandboxed judge that printed no verdict after announcing the solution's start, or ended abnormally then (a signal, an exit outside 0, 1 and 2) whatever it printed before, scores the solution 0 (`solution_error`), as sessions already did, instead of no verdict.
+- **No policy zero before the policy acts:** a session verifier that times out, loses its output or is killed before the agent's first action has no verdict; those were scored as the agent's 0.
+- **Every agent, not only ours:** `run_episode` abandons the session for any agent that returns `provider_error` or `setup_error`, including one that builds its own `AgentResult`, and sets `truncated` on the grade of a run stopped at `output_limit`.
+- **Cloud credentials refused:** E2B's `AuthenticationException` and Modal's `AuthError` and `PermissionDeniedError` count as infrastructure failures (no verdict), like a 401 or 403.
+- **Python aliases under bwrap:** local sandboxes start the solution's Python by a path inside the installation (a `~/bin` or profile link goes by its target; a venv keeps its own directory).
+- **Disk the agent filled:** a session verifier upload that fails after the agent acted is its judged 0 (`policy_sandbox_failure`), like any write, unless the network or provider failed (`VerifierUploadError`).
+- **Docker and Apple transport:** a CLI that gives no answer, or a failed exec, while the daemon can't even list containers raises `SandboxUnreachableError` (infrastructure, no verdict) instead of grading the step as a timeout or an ordinary exit. While the daemon answers, it stays the command's outcome, since what ran in the container may have caused it.
+- **No silent output cuts:** a script's stdout or stderr, or a file read with `read_output_bytes`, past its bound is the solution's failure instead of a prefix that could pass. A CLI run that prints past 256 MiB stops with the new `output_limit` termination reason (scored) instead of `error`.
+- **cli_task verifier:** git objects count only if their content hashes to their name with the declared type and length, commits only if their tree exists and every entry of it parses as git's do, HEAD and `feature` must resolve to such commits, and the commit's tree must hold `hello.txt` with the right content. Forged ref, object and tree files no longer earn the git checks.
+- **No sandbox, no local run:** where neither `sandbox-exec` nor `bwrap` works, local runs raise `SandboxError` instead of running with a warning. Set `DEEPGYM_ALLOW_UNSANDBOXED=1` to run unsandboxed anyway.
+- **Provider failures get no verdict:** an agent episode that the model provider or the network ended (a cut or dropped stream, a 5xx, 429 or overloaded error, refused credentials) stops with `provider_error`, even an OpenAI response that failed with no usage. One whose harness never started the model (CLI missing or refused) stops with the new `setup_error`. Either way the session is abandoned (`Session.abandon`), so the grade has no verdict instead of a 0. Connection errors from the anthropic and openai SDKs' `httpx2` transport now count as infrastructure failures too.
+- **CLI versions:** `ClaudeCodeAgent` and `CodexAgent` accept the tested CLI release or a later one in its major version (below 1.0, its minor version), instead of only the exact release.
+- **Docs:** verifiers that can't judge must raise, not return 0. `ClaudeCodeAgent` drives the Claude Code CLI; there is no Claude Agent SDK integration.
+
 ## 0.6.0 (2026-10-23)
 
 - **Agent harnesses:** `deepgym.agents` adds `AnthropicComputerAgent`, `OpenAIComputerAgent`, `ClaudeCodeAgent` and `CodexAgent`, each driving a live session. `DeepGym.run_episode(env, agent, budget=...)` runs the episode and returns `(AgentResult, RunResult)`; the callable interface still returns `(Trajectory, RunResult)`. The Claude Code and Codex CLIs are installed at pinned versions and checked in the sandbox. See `docs/agents.md`.

deepgym/README.md

@@ -295,8 +295,9 @@ scrolls and drags in screenshot pixels. More in [computer use][computer-use].
 
 ### Run Claude Code or Codex in the sandbox
 
-The CLI runs headless inside the session's sandbox as an unprivileged user, and DeepGym
-scores the state it leaves:
+`ClaudeCodeAgent` drives the Claude Code CLI (`claude -p`), not the Claude Agent SDK. The
+CLI runs headless inside the session's sandbox as an unprivileged user, and DeepGym scores
+the state it leaves:
 
 ```python
 from deepgym import DeepGym, MultiTurnEnvironment
@@ -456,7 +457,7 @@ verl loads DeepGym's reward function with one config line (see
 | What it is | RL training framework | RL environments and verifiable rewards |
 | Does the training | Yes: rollouts on vLLM/SGLang, updates on FSDP/Megatron, PPO, GRPO, DAPO and more | No, it plugs into verl, TRL, OpenRLHF or your own loop |
 | Tasks | You bring the dataset and write the reward function | 31 built-in environments; importers for HumanEval, MBPP, EvalPlus, BigCodeBench, SWE-bench Pro, MiMo, CyberGym and Harbor datasets such as Terminal-Bench 2.0 |
-| Running model code | Your reward function, or a remote code sandbox service | Daytona, Modal, E2B, Docker, Apple container, or locally under sandbox-exec/bwrap (without them, local runs get process separation and a warning) |
+| Running model code | Your reward function, or a remote code sandbox service | Daytona, Modal, E2B, Docker, Apple container, or locally under sandbox-exec/bwrap (without them, local runs refuse unless `DEEPGYM_ALLOW_UNSANDBOXED=1`) |
 | Reward hacking | Up to your reward function | The solution runs apart from the verifier with no network, so it can't read the expected answers or print a fake score; `deepgym qa` checks that empty and stub solutions earn nothing and reward-hack ones stay below 0.5 before you train |
 | Multi-turn agents | Agent loop and tool interface | One live sandbox per episode (shell or desktop), with a verl agent loop and session tool |
 | Computer use | Not built in | Desktop tasks, scored on the state the agent leaves behind |

deepgym/docs/agents.md

@@ -5,7 +5,17 @@
 `run(session, initial, *, task, budget) -> AgentResult`. Start the Session first;
 the caller owns grading and cleanup. `DeepGym.run_episode(env, agent, budget=...)`
 does that lifecycle and returns `(AgentResult, RunResult)`. Its existing callable
-interface still returns `(Trajectory, RunResult)`.
+interface still returns `(Trajectory, RunResult)`. `ClaudeCodeAgent` and `CodexAgent`
+drive the Claude Code and Codex CLIs inside the sandbox; there is no Claude Agent SDK
+integration.
+
+An episode that stops through nothing the policy did gets no verdict, never a 0:
+`termination_reason` is `provider_error` when the model provider or the network failed (a
+dropped or cut stream, a 5xx, 429 or overloaded error, our credentials refused) and
+`setup_error` when the harness never started the model (its CLI missing or refused). The
+session is then abandoned, so `run_episode`, or `score()` under `dg.trace()`, returns
+`has_verdict=False` with that `error_type`, whatever state the policy reached. Every other
+reason is scored, including a response the model itself cut at its output limit.
 
 Computer agents need the `anthropic` or `openai` optional extra, respectively.
 Imports stay lazy. Choose a model supporting Anthropic's
@@ -141,9 +151,10 @@ its `stop_reason`, usage and `input_json`, each call's input exactly as it strea
 ## CLI isolation and its exposure
 
 CLIs must already be installed in the sandbox image; the library installs nothing.
-Supported releases are pinned in `deepgym.agents.SUPPORTED_CLIS`. Each run checks
-the installed version inside the sandbox before launching the task; unsupported
-versions are refused. The observed version is retained in result and trace
+The tested releases are in `deepgym.agents.SUPPORTED_CLIS`. Each run checks the
+installed version inside the sandbox before launching the task: the tested release or a
+later one in its major version (below 1.0, its minor version) runs; anything else is
+refused as a `setup_error`. The observed version is retained in result and trace
 provenance. Their flags must match the adapters. The environment
 must allow `bash` or `cli`, and its backend must retain a separate network
 namespace. LocalBackend and provider-only network blocking without that namespace
@@ -223,13 +234,14 @@ environments can retain it.
 ## Upgrading the CLIs
 
 `deepgym.agents.SUPPORTED_CLIS` is the sole source of supported versions and npm
-package names. Phase 1 supports one exact release per CLI. Image builders can
-generate the installation command without maintaining their own pins:
+package names. The tested release is the oldest supported one; later compatible releases
+run too. Image builders can generate the installation command for the tested releases
+without maintaining their own pins:
 
 ```python
 from deepgym.agents import SUPPORTED_CLIS, cli_install_command
 
-print(cli_install_command())  # npm install -g, with both packages pinned
+print(cli_install_command())  # npm install -g, both tested releases
 print({name: release.version for name, release in SUPPORTED_CLIS.items()})
 ```
 

deepgym/docs/authoring/verifiers.md

@@ -34,9 +34,12 @@ Never import or run the solution yourself. Use the helpers in `deepgym_solution`
   `subprocess.CompletedProcess`; check its `returncode`. Locally it can write only inside
   `cwd`; in a sandbox it runs as the solver user, which can still write world-writable
   paths.
-  Pass `allow_subprocess=True` if the script needs to start other programs (macOS blocks
-  forking otherwise), and `loopback_port=port` to let it reach a server the verifier runs
-  on `127.0.0.1:port` (nothing else on the network).
+  Pass `allow_subprocess=True` if the script needs to start other programs. Locally on
+  macOS that is refused (the run ends with no verdict), since nothing there contains a
+  child that detaches: run such environments on Linux or a sandbox provider. Pass
+  `loopback_port=port` to let it reach a server the verifier runs on `127.0.0.1:port`
+  (nothing else on the network). Output past 64 MiB, printed or read back with
+  `read_output_bytes`, is the solution's failure, never cut to a prefix.
 - `read_output(path, cwd)` reads its files back and refuses symlinks.
 - `solution_source(path, name=None)` reads the source for static checks without running
   it.
@@ -51,6 +54,12 @@ verifier as a `deepgym_solution.SolutionError`, still of the builtin type the so
 error, so a solution can't dodge its score by breaking the verifier. Load test data before
 starting the solution: a verifier that crashes after that scores 0 too.
 
+When the verifier itself can't judge (a fixture it needs is missing, a service it checks
+against is down), it must raise, never return 0. Raised before it starts the solution, the
+run ends with no verdict, which trainers mask; a returned 0 is a real verdict that training
+learns from. So check what you need before starting the solution: after that, a crash is
+the solution's 0, as above.
+
 When the machine can't start the solution before any solution code has run (out of
 processes, say), these raise `LaunchError`. It isn't an `Exception`, so `except Exception`
 doesn't catch it. Let it propagate: the run ends with `sandbox_error` and no verdict. Once
@@ -67,9 +76,12 @@ makes the whole run a judged 0.
 - **Sandboxes (Daytona, Modal, E2B, Docker, Apple container):** the solution runs as a
   separate Linux user that can't read the judge's files, touch the judge's process, or
   write to its output, with no network.
-- **Locally:** it runs under `sandbox-exec` on macOS or `bwrap` on Linux, with the judge
-  directory, the deepgym package, your home directory and `/run` (daemon sockets) hidden.
-  If neither is available, you still get process separation and a warning.
+- **Locally:** it runs under `sandbox-exec` on macOS or `bwrap` on Linux and can read only
+  the operating system, the software trees a Python links against (Homebrew's, say), the
+  Python installation and its own run's files. Your home directory, temp directories
+  (another run's verifier, an answer key under `/tmp`) and daemon sockets aren't
+  readable, and no deepgym package is. Where neither sandbox works, local runs refuse
+  unless you set `DEEPGYM_ALLOW_UNSANDBOXED=1`; the solution can then read your files.
 
 What this blocks:
 

deepgym/docs/sandboxes.md

@@ -78,9 +78,15 @@ one for anything shared or untrusted.
   (run any command as its sudoer) on loopback, so on both DeepGym gives untrusted code its
   own offline network namespace; that needs util-linux in the image, and a judge run fails
   rather than run online without it.
-- **Locally:** the answer runs under `sandbox-exec` (macOS) or `bwrap` (Linux), with the
-  judge directory, the deepgym package, your home directory and `/run` (daemon sockets)
-  hidden. If neither is available, you still get process separation and a warning.
+- **Locally:** the answer runs under `sandbox-exec` (macOS) or `bwrap` (Linux) and can
+  read only the operating system, the software trees a Python links against, the Python
+  installation and its own run's files: not your home directory, temp directories
+  (another run's verifier), daemon sockets or any deepgym package, and on macOS it can't
+  signal anything outside its sandbox, its judge included. macOS has no way to contain a
+  child that detaches, so there the answer never starts subprocesses and
+  `run_solution(..., allow_subprocess=True)` ends the run with no verdict: run such
+  environments on Linux or a sandbox provider. Where neither sandbox works, local runs
+  refuse unless you set `DEEPGYM_ALLOW_UNSANDBOXED=1`.
   `DeepGym(mode='local', confine_local_judge=True)` also runs each whole judge inside the
   local sandbox, for verifiers that might run code in-process; `deepgym qa --mutants`
   uses it. Sandboxes don't nest, so the answer then runs as plain child processes that

deepgym/src/deepgym/_cli_worker.py

@@ -17,6 +17,10 @@
 import time
 from pathlib import Path
 
+# The CLI's output stream, which the host keeps whole for the trace and its events: a run
+# that passes this is stopped as output_limit, its own termination reason, never an error.
+# Far above a normal run's stream.
+OUTPUT_LIMIT_BYTES = 256 * 1024 * 1024
 _LOOPBACK_NO_PROXY = 'localhost,localhost.,127.0.0.1,::1,[::1],0:0:0:0:0:0:0:1,[0:0:0:0:0:0:0:1]'
 
 
@@ -166,7 +170,7 @@ def _supervise(directory: Path) -> None:
                 status['stop_reason'] = _stop_reason(directory, deadline)
                 if status['stop_reason'] is not None:
                     break
-                if output.tell() > 32 * 1024 * 1024:
+                if output.tell() > OUTPUT_LIMIT_BYTES:
                     status['stop_reason'] = 'output_limit'
                     break
                 time.sleep(0.02)
@@ -247,15 +251,25 @@ def _check_version(config: dict, status: dict, deadline: float) -> None:
             'interpreter is executable on the same unprivileged PATH (for example '
             '/usr/local/bin/node); do not load interactive nvm profiles.'
         )
-    elif version != expected:
+    elif not cli_version_supported(version, expected):
         status['version_error'] = (
-            f'Unsupported {command} version {version or "unknown"}; supported: {expected}. '
-            'Install the pinned release using deepgym.agents.cli_install_command(), '
-            'or bump SUPPORTED_CLIS, record versioned fixtures, run '
-            'tests/agents/test_cli_compat.py, and rebuild sandbox images.'
+            f'Unsupported {command} version {version or "unknown"}; supported: {expected} or '
+            'a later compatible release. Install one using '
+            'deepgym.agents.cli_install_command().'
         )
 
 
+def cli_version_supported(version: str | None, tested: str) -> bool:
+    """Whether *version* is *tested* or a later compatible release, which keeps the flags
+    and stream events the adapters use: the same major version, or below 1.0 the same
+    minor version, since there a minor release may break them."""
+    if version is None:
+        return False
+    have, want = (tuple(map(int, release.split('.'))) for release in (version, tested))
+    line = 1 if want[0] else 2
+    return have[:line] == want[:line] and have >= want
+
+
 def _stop_reason(directory: Path, deadline: float) -> str | None:
     marker = directory / 'stop'
     if marker.exists():