Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ Versions follow [SemVer](https://semver.org).

## [Unreleased]

- The README and the getting-started guide link to the quickstart repository. It runs two CPU benchmarks with the local forge and local compute, and needs no GitHub account or cluster (#563). The docs and the startup warning no longer say an uncontained session runs in a sandbox: it gets a scrubbed environment and its own home, but can read any file the account can.
- The advisory review covers exactly the head its triggering event named (`REVIEW_HEAD_SHA`, #462). If the PR has moved off that head, nothing is reviewed or posted. Otherwise the diff is read between fixed commits, every envelope records the reviewed `head_sha`, and the review is attached to that commit. When GitHub refuses its inline comments, the fallback is a body-only review on the same commit, never an unpinned comment (#569). The summarizer skips clean when its lenses read different heads. For a fork PR (`REVIEW_CHECKOUT_BASE=1`) the brief says the tree is the base branch and the change is only in the diff. Upgrading: envelopes without `head_sha` (from an older reviewer ref) still post; callers that do not pass REVIEW_HEAD_SHA behave as before.
- When check runs cannot be read on a token deployment, the warning now says the token cannot read them (fine-grained tokens have no checks permission) instead of asking for an App permission.

Expand Down
17 changes: 16 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,22 @@ where it co-develops our research codebases.

## Get started

Three commands and one file. You need a repo with a benchmark command, an
To see a run first, the [quickstart](https://github.com/outerloop-science/quickstart)
Comment thread
renmengye marked this conversation as resolved.
Resolved
Comment thread
renmengye marked this conversation as resolved.
Resolved
Comment thread
renmengye marked this conversation as resolved.
Resolved
runs two CPU benchmarks on your own machine, with no GitHub account and no
cluster. You need Python 3.12 or later, [uv](https://docs.astral.sh/uv/), Git
2.38 or later and an Anthropic API key:

```bash
pip install outerloop-science
Comment thread
renmengye marked this conversation as resolved.
Resolved
outerloop init --forge local --compute local --no-image --target me/quickstart \
Comment thread
renmengye marked this conversation as resolved.
Resolved
--forge-source https://github.com/outerloop-science/quickstart
outerloop start
```

Without a container the author's session runs as your user and can read any file
your account can, so give it a spend-capped key.

For your own repository: three commands and one file. You need a repo with a benchmark command, an
API key for the model that will write the code, and a Slurm cluster or one
machine with a GPU.

Expand Down
2 changes: 2 additions & 0 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ This page walks you through one Outerloop run end to end: install the package, d

The tutorial uses the simplest supported setup: everything runs on your machine. [Install](install.md) covers Slurm clusters, upgrades and the other deployment shapes.

To watch a run before you write a contract, the [quickstart repository](https://github.com/outerloop-science/quickstart) has two CPU benchmarks and runs without GitHub or a cluster; its README has the three commands.
Comment thread
renmengye marked this conversation as resolved.
Resolved

## Before you start

- Python 3.12 or later on a Linux machine, or any machine you can leave a terminal open on.
Expand Down
4 changes: 2 additions & 2 deletions docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ With `OUTERLOOP_COMPUTE=local`, or with no `sbatch` on PATH, `outerloop start` r

Local compute has no lanes, so GPU benchmarks need no GPU partition: evaluations and the author's launches run on the machine's own GPUs, shared first come first served. Launch arrays run their tasks in parallel as GPUs become available, and `OUTERLOOP_LOCAL_GPUS` overrides automatic detection through `nvidia-smi` (`0` disables GPU allocation). The pool also follows the loop's own `CUDA_VISIBLE_DEVICES`: start each deployment with its own list (for example `0,1` and `2,3`) to give it its own GPUs. The author's session itself starts with `CUDA_VISIBLE_DEVICES` empty, so an experiment that needs a GPU goes through the author's `launch` syscall, where it is recorded and metered against `gpu_hours_per_run`. Uncontained, that is a default the session could reset; containment is what keeps the GPU out of the session, which is one more reason to run with the image.

Without a container image, sessions run under the harness's own sandbox and evaluations run bare on your machine with your keys, and the loop says so once at start. The verification panel is off in this mode unless you set `OUTERLOOP_PANEL_UNCONTAINED=1`, because an uncontained judge holds a shell next to its own key file. A pull request opened without a panel says so. A Codex or Hermes author needs the image in every mode. For priorities, walltime accounting, or several machines, use Slurm.
Without a container image, sessions and evaluations run as your account on your machine, and the loop says so once at start. A session gets a scrubbed environment and its own home directory, but it can read any file your account can, keys included, so use a spend-capped key. The verification panel is off in this mode unless you set `OUTERLOOP_PANEL_UNCONTAINED=1`, because an uncontained judge holds a shell next to its own key file. A pull request opened without a panel says so. A Codex or Hermes author needs the image in every mode. For priorities, walltime accounting, or several machines, use Slurm.

### One machine as its own Slurm cluster

Expand Down Expand Up @@ -214,7 +214,7 @@ Only the local loop falls back to running uncontained on its own. On Slurm the t

### Running a cluster without a container

A Slurm cluster with no container runtime sets `OUTERLOOP_UNCONTAINED=1` in `~/.config/outerloop/.env`. Sessions and evaluations then run as local mode does without an image, but as Slurm jobs on your account. Sessions run under the harness's own sandbox. Evaluations run bare, in a throwaway tree with an allowlisted environment. The image is ignored even when one is on disk.
A Slurm cluster with no container runtime sets `OUTERLOOP_UNCONTAINED=1` in `~/.config/outerloop/.env`. Sessions and evaluations then run as local mode does without an image, but as Slurm jobs on your account. Sessions get a scrubbed environment and their own home directory but can read any file the account can. Evaluations run bare, in a throwaway tree with an allowlisted environment. The image is ignored even when one is on disk.

The local-mode rules carry over. The panel is off unless you set `OUTERLOOP_PANEL_UNCONTAINED=1`, and `start` refuses a Codex or Hermes author. A session can read any file your account can: the bot's credential, the model key files, the evaluator's tree and the kernel's state. Key secrecy and measurement integrity both weaken: the benchmark command can write its own eval directory, and on a multi-node benchmark any node can write every node's result files, so a measurement is only as trustworthy as the author whose code ran. Use it where the site already confines processes, or where that exposure is acceptable to you.

Expand Down
4 changes: 2 additions & 2 deletions docs/reference/environment.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,8 +214,8 @@ and Codex and Hermes authors need one in every mode.

### `OUTERLOOP_UNCONTAINED`

`1` runs a Slurm deployment without a container: sessions under the
harness's own sandbox, evaluations bare, as jobs on your account. The image
`1` runs a Slurm deployment without a container: sessions with a scrubbed
environment and a per-run home, evaluations bare, as jobs on your account. The image
is ignored. A session can then read any file the account can, keys
included. See [running a cluster without a container](../install.md#running-a-cluster-without-a-container).

Expand Down
8 changes: 4 additions & 4 deletions src/outerloop/tick.py
Original file line number Diff line number Diff line change
Expand Up @@ -3732,8 +3732,8 @@ def _service_spec_from_env(
panel = os.environ.get("OUTERLOOP_PANEL", "verify,review")
uncontained = uncontained_opt_in()
if uncontained or (not image_ok and local_mode()):
# No container: sessions run under the harness's own sandbox and
# evaluations run bare, with the operator's own account and keys. The
# No container: sessions run as the operator's account with a scrubbed
# environment and a per-run home, and evaluations run bare. The
# local loop falls back to this when it has no image; any other
# deployment needs OUTERLOOP_UNCONTAINED=1, so a missing image on a
# cluster never silently drops containment. The panel is off unless
Expand All @@ -3744,8 +3744,8 @@ def _service_spec_from_env(
if not _UNCONTAINED_WARNED:
_UNCONTAINED_WARNED = True
log.warning(
"%s; sessions run under the harness sandbox and evaluations run bare "
"as this account; the panel is %s; a codex or hermes author needs the "
"%s; sessions and evaluations run as this account and can read its "
"files; the panel is %s; a codex or hermes author needs the "
"image (docs/install.md, running without a container)",
"OUTERLOOP_UNCONTAINED=1: no container"
if uncontained
Expand Down
Loading