Execute programs. Learn from outcomes. Evolve the harness and models.
Darwin Agent Team
Homepage • Paper • Overview • Architecture • Results • Code Layout • Get Started • Run • 中文文档
Physical intelligence depends not only on the action model, but also on the executable system that organizes, checks, and improves its behavior.
HarnessPAI is a model- and embodiment-agnostic harness for Physical AI. It uses code as an executable and evolvable interface to coordinate perception, geometric reasoning, action primitives, and execution checks, rather than asking an action model to handle the entire task on its own.
The framework separates two timescales:
- Execute within a rollout. A fixed task program grounds targets, invokes perception and action primitives, and checks outcomes before proceeding. Once selected, the program runs without online high-level LLM deliberation.
- Evolve across rollouts. A coding agent uses execution feedback to diagnose failures, revise candidate programs, and validate them before promoting successful code into a reusable library.
- Reuse what works. Structured skill memory captures failure–repair experience for later tasks, while successful trajectories can become demonstrations for downstream model training.
- 🧩 One harness, multiple backends. Evaluated with VLAs, WAMs, grasp primitives, direct control APIs, and a PPO-trained policy across seven experimental settings.
- 🎯 Stronger execution without retraining the action model. The main experiments improve LIBERO-PRO success by 61.6 percentage points and RoboCasa Atomic-Seen success by 27.2 points over their respective base models.
- 🧠 Reusable repair experience. Skills are organized by workflow node and reference maintained implementations, rather than duplicating executable code in isolated notes.
- 🔁 From execution to training data. In a separate downstream experiment, fine-tuning π₀.₅-LIBERO on successful harness trajectories improves LIBERO-PRO success by 38.8 points.
| Component | Role |
|---|---|
| Task Memory | Stores known task descriptions and supports retrieval of previously evolved programs. |
| Structured Skill Memory | Organizes failure–repair knowledge around workflow nodes such as approach, grasp, and transport. |
| Executable Code Memory | Stores validated task programs and the implementations referenced by repair skills. |
| Primitive Library | Provides perception, geometric control, and learned action capabilities that task programs can compose. |
- Retrieve or build a task program. Reuse validated code for a known task; otherwise, use the task instruction, environment information, success condition, and relevant skills to construct a candidate.
- Run the fixed program. Coordinate grounding, motion, learned actions, and predefined checks. In the LIBERO hybrid rollout, code handles grounding, approach, and transport, while the VLA handles grasp and placement.
- Use feedback between rollouts. Inspect diagnostics and rollout videos, retrieve matching repair skills, and revise an isolated candidate. Unverified edits do not overwrite the validated code library.
- Validate and retain experience. Promote candidates that pass the required validation, and consolidate reusable repair knowledge into skill memory.
Workflow-graph extraction from demonstration videos, action-model capability assessment, and perception initialization are optional preparation steps, enabled when the task and available inputs call for them.
“Open-loop” refers to the program level, not to blind execution. The program stays fixed within a rollout, but it can still use current observations, feedback control, predefined checks, and recovery logic. Program edits happen between rollouts. Avoiding online high-level LLM deliberation does not mean avoiding perception or action-model inference.
See the paper for the full framework and evolution mechanism.
Success rates are reported in %; gains are absolute percentage points (pp). The action backends remain frozen during harness evolution.
| Benchmark / evaluated subset | Reference method | Reference | HarnessPAI | Gain |
|---|---|---|---|---|
| LIBERO — Spatial, Object, Goal | π₀.₅-LIBERO | 96.9 | 98.1 | +1.2 pp |
| LIBERO-PRO — three suites × Swap / Task | π₀.₅-LIBERO | 34.9 | 96.5 | +61.6 pp |
| RoboCasa Target50 — Atomic-Seen, 18 tasks | WorldDreamer | 65.0 | 92.2 | +27.2 pp |
| robosuite — seven manipulation tasks | ASPIRE | 81.0 | 96.3 | +15.3 pp |
Evaluation notes. LIBERO and LIBERO-PRO evaluate each task over 50 seeds, including the 15 seeds used for program evolution and 35 additional seeds. RoboCasa evaluates WorldDreamer and HarnessPAI under the same 20-seed setting. The robosuite ASPIRE baseline is taken from prior work, not a controlled same-backend ablation. See the paper for complete protocols, per-task results, and differences between reference methods.
| Setting | Embodiment / task | Backend |
|---|---|---|
| robosuite | Single-arm and bimanual manipulation | GraspNet |
| LIBERO | Language-conditioned tabletop manipulation | π₀.₅-LIBERO |
| LIBERO-PRO | Manipulation under object-position and instruction perturbations | π₀.₅-LIBERO; DreamZero in the transfer experiment |
| RoboCasa | Household tasks with a mobile manipulator | WorldDreamer |
| BEHAVIOR-1K | Household navigation and object pickup | GraspNet |
| VacuSim | Robot-vacuum cleaning and coverage | Direct control API |
| MicroDuck | Legged locomotion and drift correction | PPO-trained Walker policy |
These settings use task-appropriate metrics: manipulation success, navigation and pickup completion, cleaning coverage, or locomotion error. They are not combined into a single aggregate score.
- Cross-model transfer. A harness evolved with π₀.₅ on LIBERO-Object is applied unchanged to DreamZero. On the LIBERO-PRO Object suite, success rises from 0.6% to 73.4% under Swap and from 9.6% to 84.4% under Task, without re-evolving the harness.
- Lower online deliberation overhead. On LIBERO-Goal Task 1, serial execution is 23.1× faster under Swap and 18.4× faster under Task than the evaluated serial Harness VLA baseline. These are task-specific measurements, not an across-benchmark speedup. The fixed program incurs no online high-level LLM API charge; perception, action inference, simulation, hardware, and program development still have costs.
- Trajectories for post-training. In a separate experiment, successful harness rollouts are filtered by the environment's success check and used to fine-tune π₀.₅-LIBERO, raising LIBERO-PRO success from 34.9% to 73.7% (+38.8 pp). The result uses the best checkpoint within 30,000 training steps; per-task seeds 0–39 supply the additional training data and seeds 40–49 are held out for evaluation. The harness is not re-evolved around the improved model in this experiment.
Visit the project homepage for videos, qualitative comparisons, and additional analysis.
HarnessPAI targets repeated-task settings in which a program can be developed, validated, and reused. Its effectiveness depends on perception quality, usable action primitives, a checkable success condition, and the evolution budget. It organizes existing physical competence; it does not create missing low-level abilities or establish a universal one-shot solution to unseen tasks.
This repository holds the research code. A task is executed open-loop by a fixed program that perceives, reasons and calls a frozen action model (VLA / WAM) as one primitive among several; across rollouts the program is evolved closed-loop (Codex writes, Gemini diagnoses) and failures are distilled into a node-keyed skill library.
harnesspai/
├── primitives/ Primitive Library: perception / control / model
├── envs/ benchmark adapters (the only place a simulator is imported)
├── agents/ Codex (writer) + Gemini (diagnoser)
├── memory/ Task Memory · graph library · node-centric skills
├── code_library/ Executable Code Memory: promoted task packages
├── execution/ open-loop rollout of one code package (+ batch evaluation)
├── evolution/ closed loop + optional stages
│ └── stages/ graph (step 1) · capability (step 2) · perception (step 3)
├── feedback/ harness trajectories → post-training data
└── infra/ micromamba envs + model services
configs/ all configuration: envs · services · models · uv.toml · benchmarks/*.yaml
(local.yaml = keys/machine overrides, gitignored; libero/config.yaml generated)
environments/ one micromamba spec per env (+ post-install script)
third_party/ cap-x (upstream capgym/cap-x, unmodified), openpi (π0.5),
robocasa + robocasa_robosuite — git submodules
data/libero/ LIBERO-PRO bddl/init files (+ datasets symlink)
runs/ stage artifacts: runs/<bench>/<suite>/{0_expert,1_capability,2_perception,3_evolution,4_eval}
The project homepage (index.html, script.js, styles.css, assets/) lives at
the repository root and is served by GitHub Pages; _config.yml keeps the code out
of the published site.
git clone --recurse-submodules https://github.com/Darwin-Agent/HarnessPAI.git
cd HarnessPAIIf you cloned without --recurse-submodules, initialise only what is needed
(b1k / verl / curobo / robotwin inside cap-x are not used):
git submodule update --init third_party/cap-x third_party/openpi
# only for the robocasa env:
git submodule update --init third_party/robocasa third_party/robocasa_robosuite
git -C third_party/cap-x submodule update --init \
capx/third_party/LIBERO-PRO \
capx/third_party/robosuite \
capx/third_party/libero_dependencies/robosuite \
capx/third_party/sam3 \
capx/third_party/contact_graspnet_pytorchthird_party/cap-x is the official capgym/cap-x
and is never edited: everything HarnessPAI needs beyond it (gateway LLM client,
per-GPU SAM3 endpoints, LIBERO runtime helpers) lives in harnesspai/.
Large data not in git (link, don't re-download):
| What | Where it goes | Source |
|---|---|---|
| LIBERO-PRO datasets (~32 GB) | data/libero/datasets (symlink, made by environments/libero.post.sh) |
libero_datasets_src in configs/envs.yaml |
| RoboCasa kitchen assets (~23 GB) | linked by environments/robocasa.post.sh |
$ROBOCASA_ASSETS_SRC |
| Model weights (Molmo2-8B, π0.5, SAM3 HF cache) | models_dir in configs/services.yaml |
shared model dir |
Every benchmark and every model service has its own micromamba env, because
their dependencies conflict (three robosuite versions; vllm/torch 2.9 vs
sam3/torch 2.8 vs openpi/JAX). micromamba creates each env with its own Python;
packages are then installed into that Python with uv pip install --python <env>/bin/python (uv is only the installer, it creates no venv).
| Env | Python | Role | Contents | Built by |
|---|---|---|---|---|
core |
3.11 | control | harnesspai, capx (no deps), openai; evolution loop / agents / memory — no simulator, no torch | core.post.sh |
libero |
3.12 | benchmark | LIBERO-PRO, robosuite 1.4.0, contact-graspnet, pyroki, openpi-client, harnesspai | libero.post.sh |
robosuite |
3.10 | benchmark | robosuite 1.5.1, contact-graspnet, pyroki, openpi-client, harnesspai | robosuite.post.sh |
robocasa |
3.11 | benchmark (simulator only, no code targets it yet) | robocasa + robosuite 1.5.2 fork; capx/harnesspai not installed (numpy 2.2 / torch 2.7 conflict) | robocasa.post.sh |
sam3 |
3.10 | service | torch 2.8, sam3, fastapi/uvicorn | sam3.post.sh |
molmo |
3.10 | service | vllm 0.15 (torch 2.9) | molmo.post.sh |
pi05 |
3.11 | service | official openpi, installed from its own uv.lock |
pi05.post.sh |
graspnet |
3.10 | service | torch 2.8, contact_graspnet_pytorch, capx (no deps) | graspnet.post.sh |
micromamba(≥ 2.0) anduvonPATH(or in~/.local/bin).- Optional proxy for PyPI. GitHub is fetched directly (
setup.shadds it toNO_PROXY; the proxy resets long git TLS connections).
export http_proxy=http://<proxy-host>:<port> https_proxy=$http_proxy HTTP_PROXY=$http_proxy HTTPS_PROXY=$http_proxyenvironments/setup.sh core # one env
environments/setup.sh libero sam3 # several
environments/setup.sh all # every spec in environments/What setup.sh does for <name>:
micromamba create -p $MAMBA_ROOT_PREFIX/envs/<name> -f environments/<name>.yml— Python + pip only, from the conda-forge mirror in$CONDA_CHANNEL(default: Tsinghua; conda.anaconda.org is unreliable through the proxy).- Runs
environments/<name>.post.sh, which installs packages withuv pip install --python <env python>(retried up to 4×) and, forlibero/robosuite/robocasa, writes an activation hook settingMUJOCO_GL=egl,PYOPENGL_PLATFORM=eglandLIBERO_CONFIG_PATH.
Settings (all optional):
| Variable | Default | Meaning |
|---|---|---|
HARNESSPAI_MAMBA_ROOT |
mamba_root in configs/envs.yaml |
micromamba root for HarnessPAI envs (separate from ~/micromamba, so names like sam3 never collide) |
CONDA_CHANNEL |
conda_channel in configs/envs.yaml |
channel for the Python itself |
ROBOCASA_ASSETS_SRC |
robocasa_assets_src in configs/envs.yaml |
where robocasa.post.sh links kitchen assets from |
Pinned versions: capx's own dependency specifiers are loose, and an unpinned
resolve pulls releases that break at import time (torch 2.14, setuptools ≥ 80
without pkg_resources, a scipy too new for numpy 1.26, the GUI opencv-python
shadowing the headless build). So:
environments/capx-overrides.txtmirrors capx's[tool.uv] override-dependencies(numpy 1.26.4, jax < 0.4.30, headless OpenCV only, …), passed with--override.environments/constraints/{libero,robosuite}.txtfreeze the last known-good resolution (from the cap-x.venv-libero/.venv), passed with--constraint. Local version tags (+cu128) are stripped; the jax pins of the old.venvare dropped because pyroki needs jax < 0.4.30.sam3/graspnetpinsetuptools<80;graspnetbuilds contact_graspnet without build isolation (itssetup.pyimports numpy).
Proxy: PyPI and the conda mirror go through http_proxy; GitHub (pyroki, lerobot,
dlimp git dependencies) is fetched directly, because the proxy resets long git TLS
connections.
Rebuilding: setup.sh reuses an existing env and only re-runs the post-install;
delete <root>/envs/<name> to rebuild from scratch. It refuses to touch an env it
did not create (marker file .harnesspai-env); an env whose create was interrupted
before the marker was written is adopted if its micromamba history shows it was
created from the same spec.
harnesspai env check on this machine (2026-10-21) — every env except robocasa
passes:
[PASS] core [PASS] libero [PASS] robosuite [PASS] sam3
[PASS] molmo [PASS] pi05 [PASS] graspnet [FAIL] robocasa (not built)
Beyond the import check: libero resets a LIBERO-PRO environment, robosuite
resets Lift, launch_sam3_server.py and launch_contact_graspnet_server.py
import in their envs, molmo reports vllm 0.15 + CUDA, pi05 reports openpi + JAX
on GPU with the pi05_libero checkpoint cached, and the test suite passes in
core. The servers themselves have not been started (that occupies GPUs).
micromamba run -p <root>/envs/core harnesspai env list # which envs exist
micromamba run -p <root>/envs/core harnesspai env check # import-check every envconfigs/envs.yaml lists, per env, the modules env check imports.
LIBERO paths come from configs/libero/config.yaml via LIBERO_CONFIG_PATH
(written by libero.post.sh); ~/.libero is not used.
Each service runs in its own env, one instance per GPU, port = base + 100 × gpu
(configs/services.yaml). Logs and pid files go to logs/services/<name>/.
| Service | Env | Port (gpu 0) | Health |
|---|---|---|---|
sam3 |
sam3 |
8114 | /docs |
molmo |
molmo |
8122 | /v1/models |
pi05 |
pi05 |
8000 (single instance) | /healthz |
graspnet |
graspnet |
8115 | /docs |
harnesspai serve up sam3 molmo --gpus 0,2 --wait
harnesspai serve up pi05 --wait
harnesspai serve status
harnesspai serve down sam3 --gpus 0,2scripts/start_*.sh query every visible GPU with nvidia-smi, sort by free memory
then utilization, and start one instance per selected GPU:
scripts/start_molmo.sh # Molmo2/vLLM on the 7 idlest GPUs
scripts/start_sam.sh # SAM3 on the 7 idlest GPUs
scripts/start_pyroki.sh # PyRoKi IK server on one free GPU
scripts/start_pi05.sh # pi0.5 policy server on one free GPUArguments and overrides:
| How | Example |
|---|---|
| explicit GPUs (positional) | scripts/start_sam.sh 0,2,3 |
| env | GPU_LIST=0,2 scripts/start_sam.sh |
| GPs count / wait / free-memory floor | N=4 WAIT=0 MIN_FREE_MIB=60000 scripts/start_pi05.sh |
If the host exposes fewer GPUs than requested the scripts warn and use all of
them (--strict on the GPU picker turns that into an error). Indices are mapped
back through CUDA_VISIBLE_DEVICES when it is set. Bare python -m harnesspai.infra.gpu lists the GPUs with free memory and utilization.
GPU_LIST also matters for consistency: a rollout on GPU g talks to the
instance on GPU g through SAM3_SERVICE_URL / MOLMO_SERVICE_URL
(base + 100 × g), so start at least as many service instances as rollout GPUs.
Models, gateway URL and proxy bypass live in configs/models.yaml. The Gemini
diagnoser needs a key: vlm.api_keys in configs/local.yaml, or API_KEY (several
comma-separated in API_KEYS for rotation). Codex uses coder.api_key in
configs/local.yaml, else ~/.codex/config.toml.
All commands run in the core env (except graph, assess and perceive, see Steps 1-3 below); rollouts are launched in the benchmark env
named by configs/benchmarks/<bench>.yaml.
Language of terminal output, --help and the human-readable reports (REPORT.md,
phase_report.md, perception README.md): harnesspai --lang en|zh <cmd> ..., or
HARNESSPAI_LANG=en|zh, or lang: en|zh in configs/local.yaml (default zh). Prompts
and the run artifacts agents read back (transcript, trace, round_report.md, ...) are
always English.
# optional step 1: expert video -> workflow graph / graph-guided expert notes
harnesspai graph --suite libero_object --task-id 0
harnesspai expert --suite libero_object --task-id 0
# optional step 2: VLA capability (rollouts, then per-phase good/needs_code/mixed)
harnesspai assess run --benchmark libero_object
harnesspai assess perturb --benchmark libero_object
harnesspai assess phase --eval-dir runs/libero/libero_object/1_capability --graph-dir harnesspai/memory/graph/data/libero/libero_object
# optional step 3: perception module
harnesspai perceive --bench libero_object --task-id 0
# step 4: main loop
harnesspai evolve "pick up the alphabet soup and place it in the basket" \
--target <task package>/planning.py --task-module <module> \
--graph harnesspai/memory/graph/data/libero/libero_object/task0.json \
--suite libero_object --task-pkg task0 --task-id 0 \
--eval-suites libero_object,libero_object_swap,libero_object_task \
--episode-start 50 --episode-end 65 [--dry-run]Task packages live only in harnesspai/code_library/ (scaffolded, evolved and
promoted there); nothing is resolved from cap-x.
Runs in the libero env (rendering the expert *_demo.hdf5 needs libero + h5py +
imageio). Gemini watches the expert video and returns a phase list; Codex turns it
plus a few keyframes into the graph JSON, which is validated and written to the
graph library.
cd <repo>
export API_KEY=sk-... # Gemini key (or vlm.api_keys in configs/local.yaml)
HarnessPAI=<mamba_root> # configs/envs.yaml
# one task
micromamba run -p $HarnessPAI/envs/libero harnesspai graph --suite libero_goal --task-id 0
# whole suite (task0-9) + graphs.md
micromamba run -p $HarnessPAI/envs/libero harnesspai graph --suite libero_goal
# any mp4 (non-LIBERO)
micromamba run -p $HarnessPAI/envs/libero harnesspai graph --video /path/expert.mp4 \
--task "lift the cube" --benchmark robosuite --suite lift --task-id 0| Output | Path |
|---|---|
| graph | harnesspai/memory/graph/data/<bench>/<suite>/taskN.json (e.g. libero/libero_goal/task0.json) |
| mermaid overview (whole suite) | harnesspai/memory/graph/data/<bench>/<suite>/graphs.md |
| intermediates (expert mp4, keyframes, Gemini phases, Codex workdir) | runs/_work/graph/<bench>/taskN/ |
Options: --view agentview|eye_in_hand, --max-frames 40 (frames sampled for
Gemini), --n-keyframes 6 (keyframes for Codex), --model (override the Codex
model), --out <path> (write elsewhere instead of the library), --quiet.
Re-running overwrites the library's taskN.json; the graphs are tracked in git, so
git checkout restores them. Use --out /tmp/task0.json to compare first.
Runs in the libero env with the VLA server up. It measures where the VLA
(default π0.5) succeeds or fails on each graph action phase, so the main loop only
asks Codex to write code for the phases the VLA cannot do. Uses the graphs from
Step 1.
cd <repo>
export API_KEY=sk-... # Gemini key (phase tracing)
HarnessPAI=<mamba_root> # configs/envs.yaml
SUITE=libero_goal
# 0. VLA server (port 8000)
micromamba run -p $HarnessPAI/envs/core harnesspai serve up pi05 --wait
# 1. base suite: classify -> rollout -> aggregate (skips stages already done)
micromamba run -p $HarnessPAI/envs/libero harnesspai assess run --benchmark $SUITE
# 2. LIBERO-PRO perturbations (swap / task), same episode window as the base
# (needs the 100-state .init; generate it once if missing:
# python -m harnesspai.envs.libero.init_states --suite ${SUITE}_swap)
micromamba run -p $HarnessPAI/envs/libero harnesspai assess perturb --benchmark $SUITE
# 3. per-phase verdict: good / needs_code / mixed
micromamba run -p $HarnessPAI/envs/libero harnesspai assess phase \
--eval-dir runs/libero/$SUITE/1_capability \
--graph-dir harnesspai/memory/graph/data/libero/$SUITE| Sub-step | What it does | Output (under runs/libero/<suite>/1_capability/) |
|---|---|---|
classify (in run) |
Codex labels every graph action node keep / drop | harnesspai/evolution/stages/capability/node_policy_<bench>_<suite>.json |
rollout (in run) |
VLA rollouts on the base suite | base_test/taskN/results.jsonl, base_test/taskN/epM/{run.log, success.mp4|failure.mp4} |
aggregate (in run) |
success rates + graph split | _summary.json |
perturb |
the same rollouts on <suite>_swap / <suite>_task |
swap_test/, task_test/ (each with _summary.json, REPORT.md) |
phase |
Gemini traces success/failure videos onto graph phases | phase_capability/phase_summary.json, phase_report.md |
evolve --only-needs-code-phases reads phase_summary.json and gives Codex only
the needs_code phases; good / mixed phases are delegated to the VLA.
Common options:
run:--tasks 0-9,--ep-start 50 --ep-end 70(exclusive),--max-parallel 5,--policy pi05,--port 8000,--only classify|rollout|aggregate,--force(or--force-classify/--force-rollout/--force-aggregate),--no-video.perturb:--perturbations swap,task,--tasks,--ep-start 50 --ep-end 65,--force.phase:--task-id N(repeatable),--n-success 3 --n-failure 3,--max-frames 48,--mode codex|direct,--force.
Runs in the libero env with the SAM3 + Molmo fleet up (one pair per GPU in
fleet_gpus, configs/services.yaml). It produces, per task, a verified
perceive_taskN.py that localizes every instruction noun / relation with
SAM3 + Molmo only (no VLM at runtime). Step 4 calls this module as-is instead of
writing perception code. Reads the Step 2 rollout videos
(1_capability/{base,swap,task}_test) and the task's task_prompts.json in Task
Memory.
cd <repo>
export API_KEY=sk-... # Gemini key (ready-frame pick, verification)
HarnessPAI=<mamba_root> # configs/envs.yaml
SUITE=libero_object
# 0. perception fleet: SAM3 + Molmo on every fleet GPU
scripts/start_sam.sh 0,2,3,4,5,6,7
scripts/start_molmo.sh 0,2,3,4,5,6,7
# 1. (recommended) gold-standard scene labels: one labeled.png per task, used by the
# verifier as a cross-check. Run on the whole suite so round 2 (perturbed nouns)
# can borrow cross-task references.
micromamba run -p $HarnessPAI/envs/libero \
python -m harnesspai.evolution.stages.perception.scene_label --suite $SUITE
# 2. perception evolution (base + swap + task episodes jointly)
micromamba run -p $HarnessPAI/envs/libero harnesspai perceive --bench $SUITE --task-id 0
micromamba run -p $HarnessPAI/envs/libero harnesspai perceive --bench $SUITE --tasks 0-9Per task:
- parse the instruction into
{nouns, relations}and seedprompts.json; - for every episode in
[ep-start, ep-end)Gemini picks the "ready" frame from the first--n-candidatesframes of the rollout mp4; - loop for at most
--max-roundsrounds: runperceive_taskN.pyon every ready frame (episodes spread over the fleet GPUs) → Gemini judges every noun / relation pass/fail on the overlay → if any item fails, Codex editsperceive_taskN.py(syntax-checked, rolled back on error) from the failures; - converged = every item passes on every episode of every variant; Codex then
generalizes the code into a skill and a per-task
README.mdis written.
| Output | Path |
|---|---|
| scene labels | runs/<bench>/<suite>/2_perception/scene_label/taskN/{labeled,ready_frame,candidates}.png |
| perception code (main artifact) | runs/<bench>/<suite>/2_perception/taskN/perceive_taskN.py |
| query phrasings / notes | .../taskN/prompts.json, .../taskN/notes.md (optional, hand-written) |
| per-round log and verdicts | .../taskN/evolution.log, round{N}.json |
| per-episode result + overlay | .../taskN/perception/, ready frames in .../taskN/ready_frames/ |
| convergence marker | .../taskN/converged.json (a converged task is skipped on re-run) |
| suite report | runs/<bench>/<suite>/2_perception/REPORT.md |
| generalized skill | harnesspai/memory/skill/data/perception/taskN_strategy.py |
Options: --bench libero_10|libero_goal|libero_object|libero_spatial (_swap /
_task suffix with --no-joint runs one variant only, output under
2_perception/{swap,task}/), --tasks 0-9 / --task-id N, --ep-start 50 --ep-end 66 (exclusive), --max-rounds 6, --n-candidates 10, --max-workers N
(use only the first N fleet GPUs), --codex-model, --rollouts-dir, --out-dir,
--force (ignore converged.json and reseed the prompts).
Runs in the core env; every rollout is launched in the benchmark env from
configs/benchmarks/<bench>.yaml. Needs the VLA server (π0.5) and, if the task
code uses perception, the SAM3 + Molmo services on the rollout GPUs. Steps 1-3 are
optional inputs: the graph (Step 1) is required, phase_summary.json (Step 2) is
used with --only-needs-code-phases, and 2_perception/taskN/perceive_taskN.py
(Step 3) is injected automatically when it exists (else prompts.json wording).
Each round works on an isolated candidate copy; the code library is changed only after a full successful batch:
| Step | What |
|---|---|
| ① Write | Codex edits the target files (round 0 from the task, later rounds from the ③④ instruction) |
| ② preflight | python -m <task module> --help + task identity; round 0 adds a one-episode probe; failures go back to the writer in the same round (--smoke-retries) |
| ①.6 Probe | re-run up to 3 episodes that failed last round and fix until they pass (--probe-max-attempts) |
| ② Execute | open-loop rollout of every episode × every suite in --eval-suites |
| ②.5 Diagnose | Gemini watches the rollouts + graph against a baseline video (base suite: --expert-hdf5; swap/task: a success.mp4 from Step 2) |
| ③④ Synthesize | Codex read-only self-review → next round's instruction |
Converged = every episode of every suite succeeds (a reviewer saying CONVERGED
does not stop the loop). The candidate is then promoted into
harnesspai/code_library/<bench>/<suite>/<task_pkg>/ and the run's failure→fix
pairs are written to the skill library (--no-skill-writeback disables it). At
least 5 rounds are budgeted (--rounds, default 10).
cd <repo>
export API_KEY=sk-... # Gemini (②.5)
HarnessPAI=<mamba_root> # configs/envs.yaml
PKG=harnesspai/code_library/libero/object/task0
micromamba run -p $HarnessPAI/envs/core harnesspai serve up pi05 --wait
# check the plan and the run dir first (no agents, no rollout)
micromamba run -p $HarnessPAI/envs/core harnesspai evolve \
"pick up the alphabet soup and place it in the basket" \
--target $PKG/planning.py --task-module harnesspai.code_library.libero.object.task0 \
--graph harnesspai/memory/graph/data/libero/libero_object/task0.json \
--suite libero_object --task-pkg task0 --dry-run
# real run: base + swap + task, episodes 50..64, 4 rollout GPUs
micromamba run -p $HarnessPAI/envs/core harnesspai evolve \
"pick up the alphabet soup and place it in the basket" \
--target $PKG/planning.py --task-module harnesspai.code_library.libero.object.task0 \
--graph harnesspai/memory/graph/data/libero/libero_object/task0.json \
--suite libero_object --task-pkg task0 \
--eval-suites libero_object,libero_object_swap,libero_object_task \
--episode-start 50 --episode-end 65 --eval-gpus 0,2,3,4 \
--expert-hdf5 <LIBERO-PRO demo .hdf5> --only-needs-code-phasesMissing target files are scaffolded in the candidate (no sibling task code is
copied); --target is repeatable. --task-id defaults to the number in
--task-pkg.
Output (under runs/<bench>/<suite>/3_evolution/<task_pkg>__<timestamp>/) |
Contents |
|---|---|
transcript.md |
the run's story line |
run_meta.json, success_contract.md |
metadata; the success contract every round is judged against |
candidate/ |
the isolated working copy Codex edits |
runs/round_NNN/codex_edit.md, diff.md, code/ |
what Codex wrote this round |
runs/round_NNN/rollout*/, probe_*/, trace.md, stderr.log |
rollout and probe results |
runs/round_NNN/video_diagnosis.md, codex_analysis.md, round_report.md |
②.5 diagnosis, ③④ review, human-readable round summary |
final/diff.md |
final diff against the starting targets |
Common options: --rounds 10, --eval-parallel N (episodes per GPU),
--resume-from <run dir> (continue an unfinished run's candidate in a new run),
--codex-model, --codex-timeout 1800 / --codex-synth-timeout 1800,
--no-video-diagnosis, --vdm-model / --vdm-url / --vdm-key,
--vla-eval-dir (default 1_capability/phase_capability), --benchmark
(default: inferred from the --graph path), --run-dir, --no-verbose.
harnesspai evolve converges on one episode window; this runs the same package
over a wider window, with every selected GPU carrying several simulators at once
(one episode = one process, so --per-gpu 5 really means five LIBERO sims per
GPU). Workers on GPU g talk to the SAM3/Molmo instance on GPU g
(base + 100·g, see §3), so start as many service instances as rollout GPUs.
# 7 idlest GPUs × 5 envs = 35 concurrent sims, base + swap + task suites
scripts/run_eval_libero.sh libero_object task0
# explicit GPUs, wider window, base suite only
SUITES=libero_object EPISODES=0-100 PER_GPU=5 \
scripts/run_eval_libero.sh libero_object task0 0,1,2,3
# evaluate a package that has not been migrated into harnesspai/code_library yet
PACKAGE=third_party/cap-x/capx/eva/code_library/libero/object/task0 \
scripts/run_eval_libero.sh libero_object task0 0,1The script picks GPUs, then calls python -m harnesspai.execution.evaluate.
Before fanning out it runs one real episode as a runtime probe — --help
only reaches argparse, so an import that fails inside run_task would otherwise
crash every worker and leave N empty episode dirs.
Output (under runs/<bench>/<suite>/4_eval/<task_pkg>__<ts>/) |
Contents |
|---|---|
plan.json |
module, package dir, python, suites, episodes, GPUs, workers |
summary.md, summary.json |
per-suite success count/rate, failed episodes, wall time |
_probe/epNNN/ |
the probe episode (stdout/stderr/exit code) |
<suite>/epNNN/ |
*.mp4, diagnostics.jsonl, phase_frames.json, stdout.log, stderr.log, exit_code |
Success is read from diagnostics.jsonl alone. Exit codes are not a signal:
the task packages disagree (libero goal/object return 1 on success, robosuite
and robocasa return 0, libero/spatial returns a success count), so a
successful episode would otherwise read as a failure.
HarnessPAI: An Evolving Harness for Physical AI
Darwin Agent Team
Read the paper · Explore the project
@article{wang2026harnesspai,
title = {{HarnessPAI}: An Evolving Harness for Physical AI},
author = {Xin Wang and Wenhao Wu and Menghao Zhang and Zhi Wang and Kun Shao and Jian Luan and Yang Li and Qing Li and Shangding Gu and Huichi Zhou and Shuqing Shi and Fei Ni and Shuo Lu and Weicheng Meng and Kang Li and Jin Wu and Kang Zhao and Shangmin Guo and Gen Li and Yongqiang Tang and Zhizhong Zhang and Yuan Xie and Heng Qu},
journal = {arXiv preprint arXiv:2609.29166},
year = {2026}
}By the Darwin Agent Team

