macOS containment: run sessions and evaluations without an image under a Seatbelt profile

CHANGELOG.md

@@ -6,10 +6,11 @@ Versions follow [SemVer](https://semver.org).
 
 ## [Unreleased]
 
+- On macOS without an image, sessions, evaluations and panel lenses run under a packaged Seatbelt profile (`sandbox-exec`, #292): they write only the workspace and a scratch directory, read only system and toolchain paths, and connect out only to TCP port 443, DNS and local ports. Linux and Apptainer runs are unchanged.
 - Kernel events now append to a per-run `trajectory.jsonl` with sequence numbers and a SHA-256 chain. `outerloop trajectory <run-id> [--verify] [--json]` reads it without changing run state. Operational files and their formats are unchanged; older kernels ignore the new file. Upgrading: runs started before this kernel have no trajectory and read as "no trajectory" until their next recorded event. No backfill is attempted.
 - Add a printable agent-text inventory and protocol wording lint, with clearer refusal recovery guidance (#487, parts 1–2).
 - `BENCHMARKS.md` shows `—` instead of "provenance unknown" in the main-commit column of a baseline row that nothing has improved on yet, as on a freshly seeded repository.
-- The README and the getting-started guide link to the quickstart repository. It runs two CPU benchmarks with the local forge and local compute, and needs no GitHub account or cluster (#563). The docs and the startup warning no longer say an uncontained session runs in a sandbox: it gets a scrubbed environment and its own home, but can read any file the account can.
+- The README and the getting-started guide link to the quickstart repository. It runs two CPU benchmarks with the local forge and local compute, and needs no GitHub account or cluster (#563). The docs and the startup warning no longer say an uncontained session runs in a sandbox: on Linux without an image, it gets a scrubbed environment and its own home, but can read any file the account can.
 - The advisory review covers exactly the head its triggering event named (`REVIEW_HEAD_SHA`, #462). If the PR has moved off that head, nothing is reviewed or posted. Otherwise the diff is read between fixed commits, every envelope records the reviewed `head_sha`, and the review is attached to that commit. When GitHub refuses its inline comments, the fallback is a body-only review on the same commit, never an unpinned comment (#569). The summarizer skips clean when its lenses read different heads. For a fork PR (`REVIEW_CHECKOUT_BASE=1`) the brief says the tree is the base branch and the change is only in the diff. Upgrading: envelopes without `head_sha` (from an older reviewer ref) still post; callers that do not pass REVIEW_HEAD_SHA behave as before.
 - When check runs cannot be read on a token deployment, the warning now says the token cannot read them (fine-grained tokens have no checks permission) instead of asking for an App permission.
 

README.md

@@ -41,8 +41,10 @@ outerloop init --forge local --compute local --no-image --target me/quickstart \
 outerloop start
 ```
 
-Without a container the author's session runs as your user and can read any file
-your account can, so give it a spend-capped key.
+On macOS, sessions and evaluations run under a Seatbelt profile: they write their
+workspace and a scratch directory, read system and toolchain paths, and cannot read
+the rest of your home. On Linux without a container they can read any file your
+account can. Either way, use a spend-capped API key.
 
 For your own repository: three commands and one file. You need a repo with a benchmark command, an
 API key for the model that will write the code, and a Slurm cluster or one
@@ -126,8 +128,8 @@ build. See [the reviewer guide](https://docs.outerloop.science/reviewer/).
   output are rendered to the agent as fenced data, not instructions, which
   lowers the risk of injected text being followed without guaranteeing it.
   A contained session gets no GitHub credential in its environment and
-  cannot read the operator's key files; an uncontained local session runs
-  as your user and can read any file you can, key files included.
+  cannot read the operator's key files. Without a container, a macOS session
+  runs under Seatbelt; a Linux session can read any file you can, key files included.
 - **Budgets in code.** Launches, GPU-hours, and runs per week are enforced
   by the kernel, not left to the agent.
 - **No model lock-in.** Claude Code, Codex, and hermes-agent are wired

docs/install.md

@@ -10,7 +10,7 @@ Outerloop is self-hosted. You run it, with your keys, on your compute, against y
 - `uv` on PATH, or in the directory its installer uses. Evaluations and launches run through `uv run`, and `outerloop start` stops with a message when it cannot find it.
 - A GitHub repository you administer.
 - An API key for the author's model, and separate keys for the judges.
-- For contained runs, Linux with [Apptainer](https://apptainer.org). macOS has no Apptainer, so runs there stay uncontained.
+- For contained runs, Linux with [Apptainer](https://apptainer.org). macOS uses Seatbelt through `/usr/bin/sandbox-exec` without a VM.
 
 The orchestrator is CPU-only and makes outbound connections only. Anywhere that can reach GitHub and your model provider works.
 
@@ -46,7 +46,7 @@ With `OUTERLOOP_COMPUTE=local`, or with no `sbatch` on PATH, `outerloop start` r
 
 Local compute has no lanes, so GPU benchmarks need no GPU partition: evaluations and the author's launches run on the machine's own GPUs, shared first come first served. Launch arrays run their tasks in parallel as GPUs become available, and `OUTERLOOP_LOCAL_GPUS` overrides automatic detection through `nvidia-smi` (`0` disables GPU allocation). The pool also follows the loop's own `CUDA_VISIBLE_DEVICES`: start each deployment with its own list (for example `0,1` and `2,3`) to give it its own GPUs. The author's session itself starts with `CUDA_VISIBLE_DEVICES` empty, so an experiment that needs a GPU goes through the author's `launch` syscall, where it is recorded and metered against `gpu_hours_per_run`. Uncontained, that is a default the session could reset; containment is what keeps the GPU out of the session, which is one more reason to run with the image.
 
-Without a container image, sessions and evaluations run as your account on your machine, and the loop says so once at start. A session gets a scrubbed environment and its own home directory, but it can read any file your account can, keys included, so use a spend-capped key. The verification panel is off in this mode unless you set `OUTERLOOP_PANEL_UNCONTAINED=1`, because an uncontained judge holds a shell next to its own key file. A pull request opened without a panel says so. A Codex or Hermes author needs the image in every mode. For priorities, walltime accounting, or several machines, use Slurm.
+Without a container image, sessions and evaluations run as your account on your machine, and the loop says so once at start. On macOS a Seatbelt profile (`/usr/bin/sandbox-exec`) confines them: they write only the workspace and a scratch directory, read only system and toolchain paths, and connect out only to TCP port 443, DNS and local ports (a local model endpoint). If `sandbox-exec` is missing, the loop warns and runs them uncontained. The profile does not open the keychain, so use an API key rather than a subscription login. On Linux they get a scrubbed environment and their own home but can read any file your account can, keys included. Use a spend-capped key. The verification panel is off in this mode unless you set `OUTERLOOP_PANEL_UNCONTAINED=1`, because an uncontained judge holds a shell next to its own key file. A pull request opened without a panel says so. A Codex or Hermes author needs the image in every mode. For priorities, walltime accounting, or several machines, use Slurm.
 
 ### One machine as its own Slurm cluster
 
@@ -208,7 +208,7 @@ GPU jobs request at least 8 cores and 64 GB of host memory per GPU. On nodes wit
 
 ## The agent image
 
-On a cluster, evals and launches run inside the Apptainer image at `OUTERLOOP_IMAGE` (default `~/outerloop-images/agent-py312.sif`), in a jail that binds only the checked-out tree. An eval that needs data must fetch it into the tree. The published image lives at [huggingface.co/outerloop-science/agent-image](https://huggingface.co/outerloop-science/agent-image), built from `containers/agent-py312.def`. On Linux with Apptainer installed, `outerloop init` downloads it (about 200 MB), verifies its published checksum and records it. `--image` points at your own, and `--no-image` keeps runs uncontained even when an image is on disk by writing `OUTERLOOP_IMAGE=`.
+On a cluster, evals and launches run inside the Apptainer image at `OUTERLOOP_IMAGE` (default `~/outerloop-images/agent-py312.sif`), in a jail that binds only the checked-out tree. An eval that needs data must fetch it into the tree. The published image lives at [huggingface.co/outerloop-science/agent-image](https://huggingface.co/outerloop-science/agent-image), built from `containers/agent-py312.def`. On Linux with Apptainer installed, `outerloop init` downloads it (about 200 MB), verifies its published checksum and records it. `--image` points at your own, and `--no-image` skips Apptainer even when an image is on disk (macOS still uses Seatbelt when available) by writing `OUTERLOOP_IMAGE=`.
 
 Only the local loop falls back to running uncontained on its own. On Slurm the tick does not service a target without an image, and its log names the setting below, so a lost image never silently drops containment. Codex and Hermes authors need the image in every mode.
 

docs/reference/environment.md

@@ -206,8 +206,9 @@ PAT rather than a GitHub App. One identity or the other, never both.
 ### `OUTERLOOP_IMAGE`
 
 The Apptainer image sessions and evals run in. An empty value is the
-off-switch for local Claude runs: they stay uncontained even when an image
-is on disk. On Slurm the tick does not service a target without an image,
+off-switch for local Claude runs: they skip Apptainer even when an image
+is on disk. Without an image, macOS runs sessions and evaluations under the packaged
+Seatbelt profile; Linux runs them uncontained. On Slurm the tick does not service a target without an image,
 and Codex and Hermes authors need one in every mode.
 
 - Default: `~/outerloop-images/agent-py312.sif`

pyproject.toml

@@ -96,6 +96,7 @@ testpaths = ["tests"]
 # the tests affected by your edits.
 addopts = "-ra --strict-markers -n auto"
 markers = [
+    "seatbelt: exercises the macOS containment wrapper (including real enforcement)",
     "real_locate_harness: exercises the real harness lookup (test_init fixture skips its patch)",
     "slow: long-running; nightly/manual only",
     "llm: calls a paid LLM API; manual only, never CI",

src/outerloop/attempt.py

@@ -33,6 +33,7 @@
 from outerloop.appauth import add_credential_args
 from outerloop.brief import BudgetState, distill_lessons
 from outerloop.compute import LocalCompute
+from outerloop.containment import seatbelt_available
 from outerloop.contract import (
     Benchmark,
     BenchmarkNotFoundError,
@@ -3833,7 +3834,7 @@ def _panel_lenses_from_args(
         prepared.append((kind, backend, model, lens_key))
     for kind, backend, model, lens_key in prepared:
         try:
-            if not args.image:
+            if not args.image and not seatbelt_available():
                 log.warning(
                     "panel lens %s runs uncontained (no image): the judge shares "
                     "this account with the operator's keys",

src/outerloop/containment/__init__.py

@@ -0,0 +1,135 @@
+"""Seatbelt containment for launches without an Apptainer image."""
+
+from __future__ import annotations
+
+import functools
+import logging
+import os
+import shutil
+import subprocess
+import sys
+from collections.abc import Iterable
+from importlib.resources import files
+from pathlib import Path
+
+SANDBOX_EXEC = Path("/usr/bin/sandbox-exec")
+_warned = False
+
+
+def seatbelt_available() -> bool:
+    return sys.platform == "darwin" and SANDBOX_EXEC.is_file()
+
+
+def executable_roots(*executables: str) -> list[Path]:
+    """Include launch prefixes as well as symlink targets (venvs need both)."""
+    roots = {Path(sys.prefix).resolve(), Path(sys.base_prefix).resolve()}
+    for executable in (*executables, sys.executable, "uv", "python", "python3", "node"):
+        located = shutil.which(executable)
+        if not located:
+            continue
+        for path in (Path(located).absolute(), Path(located).resolve()):
+            directory = path.parent
+            prefix = directory.parent
+            needs_prefix = directory.name == "bin" and (
+                path.name.startswith("python")
+                or path.name == "node"
+                or (prefix / "pyvenv.cfg").is_file()
+                or (prefix / "package.json").is_file()
+            )
+            roots.add(prefix if needs_prefix else directory)
+    uv_python = Path(
+        os.environ.get(
+            "UV_PYTHON_INSTALL_DIR",
+            str(
+                Path(os.environ.get("XDG_DATA_HOME", str(Path.home() / ".local/share")))
+                / "uv/python"
+            ),
+        )
+    )
+    if uv_python.is_dir():
+        roots.add(uv_python.resolve())
+    return _outside_home(roots)
+
+
+def _outside_home(paths: Iterable[Path]) -> list[Path]:
+    """A tool installed straight under home (~/bin/python) must not open all of home."""
+    home = Path.home().resolve()
+    resolved = {path.resolve() for path in paths}
+    return sorted(path for path in resolved if path not in (home, *home.parents))
+
+
+@functools.cache
+def _user_cache_dir() -> str:
+    """Per-user cache dir under /private/var/folders (the CLI's keychain access
+    writes there); the rest of /private/var/folders holds other runs' temp files."""
+    try:
+        out = subprocess.run(
+            ["/usr/bin/getconf", "DARWIN_USER_CACHE_DIR"],
+            capture_output=True,
+            text=True,
+            check=True,
+        ).stdout.strip()
+        return str(Path(out).resolve()) if out else "/nonexistent"
+    except (OSError, subprocess.CalledProcessError):
+        return "/nonexistent"
+
+
+def seatbelt_wrap(
+    argv: list[str],
+    *,
+    workspace: Path,
+    scratch: Path,
+    read_only: Iterable[Path],
+    read_files: Iterable[Path] = (),
+    contained: bool = False,
+) -> list[str]:
+    """Wrap an uncontained macOS launch; callers own the scratch lifetime."""
+    global _warned
+    if contained or sys.platform != "darwin":
+        return argv
+    if not seatbelt_available():
+        if not _warned:
+            logging.getLogger(__name__).warning(
+                "sandbox-exec is missing: sessions and evaluations run uncontained"
+            )
+            _warned = True
+        return argv
+    roots = [str(root) for root in _outside_home(read_only)]
+    singles = sorted({str(path.resolve()) for path in read_files})
+    rules = "\n  ".join(
+        [f'(subpath (param "READ_ONLY_{i}"))' for i in range(len(roots))]
+        + [f'(literal (param "READ_FILE_{i}"))' for i in range(len(singles))]
+    )
+    profile = (
+        files(__package__).joinpath("macos.sb").read_text().replace("; READ_ONLY_ROOTS", rules)
+    )
+    parameters = {
+        "WORKSPACE": str(workspace.resolve()),
+        "SCRATCH": str(scratch.resolve()),
+        "USER_CACHE": _user_cache_dir(),
+    }
+    parameters.update({f"READ_ONLY_{i}": root for i, root in enumerate(roots)})
+    parameters.update({f"READ_FILE_{i}": path for i, path in enumerate(singles)})
+    flags = [flag for key, value in parameters.items() for flag in ("-D", f"{key}={value}")]
+    # the profile travels in argv, so no file the sandboxed process can write
+    return [str(SANDBOX_EXEC), *flags, "-p", profile, *argv]
+
+
+def session_wrap(
+    argv: list[str],
+    workspace: Path,
+    scratch: Path,
+    env: dict[str, str],
+    read_only: Iterable[Path] = (),
+    read_files: Iterable[Path] = (),
+) -> list[str]:
+    if sys.platform == "darwin":
+        env["TMPDIR"] = str(scratch.resolve())
+        env["CLAUDE_CODE_TMPDIR"] = str(scratch.resolve())
+    return seatbelt_wrap(
+        argv,
+        workspace=workspace,
+        scratch=scratch,
+        read_only=[*executable_roots(argv[0]), *read_only],
+        read_files=read_files,
+    )

src/outerloop/containment/macos.sb

@@ -0,0 +1,18 @@
+(version 1)
+(deny default)
+(import "system.sb")
+(allow process-fork process-exec signal)
+(allow sysctl-read mach-lookup ipc-posix-shm ipc-posix-sem system-socket iokit-open)
+(allow file-read-metadata)
+(allow file-read*
+  (subpath "/usr") (subpath "/bin") (subpath "/sbin") (subpath "/System")
+  (subpath "/Library") (subpath "/opt/homebrew") (subpath "/private/etc")
+  (subpath "/private/var/db") (subpath "/dev")
+  ; READ_ONLY_ROOTS
+)
+(allow file-read* file-write*
+  (subpath (param "WORKSPACE")) (subpath (param "SCRATCH"))
+  (literal "/dev/null") (literal "/dev/tty") (subpath (param "USER_CACHE")))
+; ports, not protocols: TCP 443 anywhere, DNS, and local endpoints
+(allow network-outbound (remote tcp "*:443") (remote udp "*:53")
+  (remote tcp "localhost:*") (literal "/private/var/run/mDNSResponder"))

src/outerloop/harness.py

@@ -27,6 +27,7 @@
 from pathlib import Path
 from typing import Any, Protocol
 
+from outerloop.containment import executable_roots, session_wrap
 from outerloop.endpoint_wait import session_url as endpoint_session_url
 from outerloop.endpoints import EndpointProfile
 from outerloop.hermes_install import hermes_ready, hermes_runtime
@@ -47,10 +48,8 @@
 # What a session's environment contains — nothing else survives from the
 # parent. HOME is deliberately NOT inherited: it is redirected to a per-run
 # directory so a session cannot read key files under the real home or poison
-# other runs via ~/.claude state. (Residual risk: the filesystem itself is
-# not sandboxed — same-user absolute paths remain readable. See the threat
-# model; the bot PAT must not live on the account that runs sessions until
-# OS-level sandboxing lands.)
+# other runs via ~/.claude state. Without an image, macOS uses Seatbelt;
+# Linux still permits reads of same-user absolute paths.
 SESSION_ENV_ALLOWLIST = ("PATH", "TERM", "LANG", "LC_ALL", "TMPDIR")
 
 # Fallbacks only — every orchestrated path threads effective_limits in
@@ -798,6 +797,11 @@ def run(
             if self.vertex is not None and self.vertex.adc_file:
                 # Also mounted when an endpoint overrides Vertex authentication.
                 provision_credential_file(Path(self.vertex.adc_file))
+            if not self.container_image:
+                adc = self.vertex.adc_file if self.vertex is not None else ""
+                command = session_wrap(
+                    command, workspace, session_home, env, read_files=[Path(adc)] if adc else []
+                )
             process = subprocess.Popen(
                 command,
                 cwd=workspace,
@@ -1416,6 +1420,14 @@ def run(
                 env[f"APPTAINERENV_{key_env}"] = self.api_key
                 env["APPTAINERENV_CODEX_HOME"] = env["CODEX_HOME"]
             provision_credential_file(session_home / ".codex" / "auth.json")
+            if not self.container_image:
+                command = session_wrap(
+                    command,
+                    workspace,
+                    session_home,
+                    env,
+                    [Path(__file__).parent, *executable_roots(self.binary)],
+                )
             process = subprocess.Popen(
                 command,
                 cwd=workspace,
@@ -1788,6 +1800,10 @@ def run(
             # cwd is the per-run home, NOT the workspace: --save_sample writes
             # its trajectory JSON to cwd, and artifacts must never land in the
             # clone (they would enter the diff).
+            if not self.container_image:
+                command = session_wrap(
+                    command, workspace, session_home, env, [repo, Path(__file__).parent]
+                )
             process = subprocess.Popen(
                 command,
                 cwd=session_home,