Skip to content

About

No description, website, or topics provided.

Resources

Stars

51 stars

Watchers

2 watching

Forks

Repository files navigation

HarnessPAI

An Evolving Harness for Physical AI

Execute programs. Learn from outcomes. Evolve the harness and models.

Darwin Agent Team

Homepage Paper

Homepage • Paper • Overview • Architecture • Results • Code Layout • Get Started • Run • 中文文档

🔭 Overview

Physical intelligence depends not only on the action model, but also on the executable system that organizes, checks, and improves its behavior.

HarnessPAI is a model- and embodiment-agnostic harness for Physical AI. It uses code as an executable and evolvable interface to coordinate perception, geometric reasoning, action primitives, and execution checks, rather than asking an action model to handle the entire task on its own.

The framework separates two timescales:

  • Execute within a rollout. A fixed task program grounds targets, invokes perception and action primitives, and checks outcomes before proceeding. Once selected, the program runs without online high-level LLM deliberation.
  • Evolve across rollouts. A coding agent uses execution feedback to diagnose failures, revise candidate programs, and validate them before promoting successful code into a reusable library.
  • Reuse what works. Structured skill memory captures failure–repair experience for later tasks, while successful trajectories can become demonstrations for downstream model training.

HarnessPAI overview: seven experimental settings, four embodiment categories, and five downstream interfaces

Highlights

  • 🧩 One harness, multiple backends. Evaluated with VLAs, WAMs, grasp primitives, direct control APIs, and a PPO-trained policy across seven experimental settings.
  • 🎯 Stronger execution without retraining the action model. The main experiments improve LIBERO-PRO success by 61.6 percentage points and RoboCasa Atomic-Seen success by 27.2 points over their respective base models.
  • 🧠 Reusable repair experience. Skills are organized by workflow node and reference maintained implementations, rather than duplicating executable code in isolated notes.
  • 🔁 From execution to training data. In a separate downstream experiment, fine-tuning π₀.₅-LIBERO on successful harness trajectories improves LIBERO-PRO success by 38.8 points.

🏗️ Architecture

HarnessPAI architecture: task memory, structured skill memory, executable code memory, and a primitive library

Memory and primitives

Component Role
Task Memory Stores known task descriptions and supports retrieval of previously evolved programs.
Structured Skill Memory Organizes failure–repair knowledge around workflow nodes such as approach, grasp, and transport.
Executable Code Memory Stores validated task programs and the implementations referenced by repair skills.
Primitive Library Provides perception, geometric control, and learned action capabilities that task programs can compose.

Execute, diagnose, revise, validate

  1. Retrieve or build a task program. Reuse validated code for a known task; otherwise, use the task instruction, environment information, success condition, and relevant skills to construct a candidate.
  2. Run the fixed program. Coordinate grounding, motion, learned actions, and predefined checks. In the LIBERO hybrid rollout, code handles grounding, approach, and transport, while the VLA handles grasp and placement.
  3. Use feedback between rollouts. Inspect diagnostics and rollout videos, retrieve matching repair skills, and revise an isolated candidate. Unverified edits do not overwrite the validated code library.
  4. Validate and retain experience. Promote candidates that pass the required validation, and consolidate reusable repair knowledge into skill memory.

Workflow-graph extraction from demonstration videos, action-model capability assessment, and perception initialization are optional preparation steps, enabled when the task and available inputs call for them.

“Open-loop” refers to the program level, not to blind execution. The program stays fixed within a rollout, but it can still use current observations, feedback control, predefined checks, and recovery logic. Program edits happen between rollouts. Avoiding online high-level LLM deliberation does not mean avoiding perception or action-model inference.

See the paper for the full framework and evolution mechanism.

📊 Results

Main task-completion results

Success rates are reported in %; gains are absolute percentage points (pp). The action backends remain frozen during harness evolution.

Benchmark / evaluated subset Reference method Reference HarnessPAI Gain
LIBERO — Spatial, Object, Goal π₀.₅-LIBERO 96.9 98.1 +1.2 pp
LIBERO-PRO — three suites × Swap / Task π₀.₅-LIBERO 34.9 96.5 +61.6 pp
RoboCasa Target50 — Atomic-Seen, 18 tasks WorldDreamer 65.0 92.2 +27.2 pp
robosuite — seven manipulation tasks ASPIRE 81.0 96.3 +15.3 pp

Evaluation notes. LIBERO and LIBERO-PRO evaluate each task over 50 seeds, including the 15 seeds used for program evolution and 35 additional seeds. RoboCasa evaluates WorldDreamer and HarnessPAI under the same 20-seed setting. The robosuite ASPIRE baseline is taken from prior work, not a controlled same-backend ablation. See the paper for complete protocols, per-task results, and differences between reference methods.

Seven experimental settings

Setting Embodiment / task Backend
robosuite Single-arm and bimanual manipulation GraspNet
LIBERO Language-conditioned tabletop manipulation π₀.₅-LIBERO
LIBERO-PRO Manipulation under object-position and instruction perturbations π₀.₅-LIBERO; DreamZero in the transfer experiment
RoboCasa Household tasks with a mobile manipulator WorldDreamer
BEHAVIOR-1K Household navigation and object pickup GraspNet
VacuSim Robot-vacuum cleaning and coverage Direct control API
MicroDuck Legged locomotion and drift correction PPO-trained Walker policy

These settings use task-appropriate metrics: manipulation success, navigation and pickup completion, cleaning coverage, or locomotion error. They are not combined into a single aggregate score.

Beyond the main results

  • Cross-model transfer. A harness evolved with π₀.₅ on LIBERO-Object is applied unchanged to DreamZero. On the LIBERO-PRO Object suite, success rises from 0.6% to 73.4% under Swap and from 9.6% to 84.4% under Task, without re-evolving the harness.
  • Lower online deliberation overhead. On LIBERO-Goal Task 1, serial execution is 23.1× faster under Swap and 18.4× faster under Task than the evaluated serial Harness VLA baseline. These are task-specific measurements, not an across-benchmark speedup. The fixed program incurs no online high-level LLM API charge; perception, action inference, simulation, hardware, and program development still have costs.
  • Trajectories for post-training. In a separate experiment, successful harness rollouts are filtered by the environment's success check and used to fine-tune π₀.₅-LIBERO, raising LIBERO-PRO success from 34.9% to 73.7% (+38.8 pp). The result uses the best checkpoint within 30,000 training steps; per-task seeds 0–39 supply the additional training data and seeds 40–49 are held out for evaluation. The harness is not re-evolved around the improved model in this experiment.

Visit the project homepage for videos, qualitative comparisons, and additional analysis.

Scope

HarnessPAI targets repeated-task settings in which a program can be developed, validated, and reused. Its effectiveness depends on perception quality, usable action primitives, a checkable success condition, and the evolution budget. It organizes existing physical competence; it does not create missing low-level abilities or establish a universal one-shot solution to unseen tasks.

🗂️ Code Layout

This repository holds the research code. A task is executed open-loop by a fixed program that perceives, reasons and calls a frozen action model (VLA / WAM) as one primitive among several; across rollouts the program is evolved closed-loop (Codex writes, Gemini diagnoses) and failures are distilled into a node-keyed skill library.

harnesspai/
├── primitives/      Primitive Library: perception / control / model
├── envs/            benchmark adapters (the only place a simulator is imported)
├── agents/          Codex (writer) + Gemini (diagnoser)
├── memory/          Task Memory · graph library · node-centric skills
├── code_library/    Executable Code Memory: promoted task packages
├── execution/       open-loop rollout of one code package (+ batch evaluation)
├── evolution/       closed loop + optional stages
│   └── stages/      graph (step 1) · capability (step 2) · perception (step 3)
├── feedback/        harness trajectories → post-training data
└── infra/           micromamba envs + model services
configs/             all configuration: envs · services · models · uv.toml · benchmarks/*.yaml
                     (local.yaml = keys/machine overrides, gitignored; libero/config.yaml generated)
environments/        one micromamba spec per env (+ post-install script)
third_party/         cap-x (upstream capgym/cap-x, unmodified), openpi (π0.5),
                     robocasa + robocasa_robosuite — git submodules
data/libero/         LIBERO-PRO bddl/init files (+ datasets symlink)
runs/                stage artifacts: runs/<bench>/<suite>/{0_expert,1_capability,2_perception,3_evolution,4_eval}

The project homepage (index.html, script.js, styles.css, assets/) lives at the repository root and is served by GitHub Pages; _config.yml keeps the code out of the published site.

1. Get the code

git clone --recurse-submodules https://github.com/Darwin-Agent/HarnessPAI.git
cd HarnessPAI

If you cloned without --recurse-submodules, initialise only what is needed (b1k / verl / curobo / robotwin inside cap-x are not used):

git submodule update --init third_party/cap-x third_party/openpi
# only for the robocasa env:
git submodule update --init third_party/robocasa third_party/robocasa_robosuite
git -C third_party/cap-x submodule update --init \
    capx/third_party/LIBERO-PRO \
    capx/third_party/robosuite \
    capx/third_party/libero_dependencies/robosuite \
    capx/third_party/sam3 \
    capx/third_party/contact_graspnet_pytorch

third_party/cap-x is the official capgym/cap-x and is never edited: everything HarnessPAI needs beyond it (gateway LLM client, per-GPU SAM3 endpoints, LIBERO runtime helpers) lives in harnesspai/.

Large data not in git (link, don't re-download):

What Where it goes Source
LIBERO-PRO datasets (~32 GB) data/libero/datasets (symlink, made by environments/libero.post.sh) libero_datasets_src in configs/envs.yaml
RoboCasa kitchen assets (~23 GB) linked by environments/robocasa.post.sh $ROBOCASA_ASSETS_SRC
Model weights (Molmo2-8B, π0.5, SAM3 HF cache) models_dir in configs/services.yaml shared model dir

2. Environments (micromamba)

Every benchmark and every model service has its own micromamba env, because their dependencies conflict (three robosuite versions; vllm/torch 2.9 vs sam3/torch 2.8 vs openpi/JAX). micromamba creates each env with its own Python; packages are then installed into that Python with uv pip install --python <env>/bin/python (uv is only the installer, it creates no venv).

Env Python Role Contents Built by
core 3.11 control harnesspai, capx (no deps), openai; evolution loop / agents / memory — no simulator, no torch core.post.sh
libero 3.12 benchmark LIBERO-PRO, robosuite 1.4.0, contact-graspnet, pyroki, openpi-client, harnesspai libero.post.sh
robosuite 3.10 benchmark robosuite 1.5.1, contact-graspnet, pyroki, openpi-client, harnesspai robosuite.post.sh
robocasa 3.11 benchmark (simulator only, no code targets it yet) robocasa + robosuite 1.5.2 fork; capx/harnesspai not installed (numpy 2.2 / torch 2.7 conflict) robocasa.post.sh
sam3 3.10 service torch 2.8, sam3, fastapi/uvicorn sam3.post.sh
molmo 3.10 service vllm 0.15 (torch 2.9) molmo.post.sh
pi05 3.11 service official openpi, installed from its own uv.lock pi05.post.sh
graspnet 3.10 service torch 2.8, contact_graspnet_pytorch, capx (no deps) graspnet.post.sh

Prerequisites

  • micromamba (≥ 2.0) and uv on PATH (or in ~/.local/bin).
  • Optional proxy for PyPI. GitHub is fetched directly (setup.sh adds it to NO_PROXY; the proxy resets long git TLS connections).
export http_proxy=http://<proxy-host>:<port> https_proxy=$http_proxy HTTP_PROXY=$http_proxy HTTPS_PROXY=$http_proxy

Build

environments/setup.sh core            # one env
environments/setup.sh libero sam3     # several
environments/setup.sh all             # every spec in environments/

What setup.sh does for <name>:

  1. micromamba create -p $MAMBA_ROOT_PREFIX/envs/<name> -f environments/<name>.yml — Python + pip only, from the conda-forge mirror in $CONDA_CHANNEL (default: Tsinghua; conda.anaconda.org is unreliable through the proxy).
  2. Runs environments/<name>.post.sh, which installs packages with uv pip install --python <env python> (retried up to 4×) and, for libero/robosuite/robocasa, writes an activation hook setting MUJOCO_GL=egl, PYOPENGL_PLATFORM=egl and LIBERO_CONFIG_PATH.

Settings (all optional):

Variable Default Meaning
HARNESSPAI_MAMBA_ROOT mamba_root in configs/envs.yaml micromamba root for HarnessPAI envs (separate from ~/micromamba, so names like sam3 never collide)
CONDA_CHANNEL conda_channel in configs/envs.yaml channel for the Python itself
ROBOCASA_ASSETS_SRC robocasa_assets_src in configs/envs.yaml where robocasa.post.sh links kitchen assets from

Pinned versions: capx's own dependency specifiers are loose, and an unpinned resolve pulls releases that break at import time (torch 2.14, setuptools ≥ 80 without pkg_resources, a scipy too new for numpy 1.26, the GUI opencv-python shadowing the headless build). So:

  • environments/capx-overrides.txt mirrors capx's [tool.uv] override-dependencies (numpy 1.26.4, jax < 0.4.30, headless OpenCV only, …), passed with --override.
  • environments/constraints/{libero,robosuite}.txt freeze the last known-good resolution (from the cap-x .venv-libero / .venv), passed with --constraint. Local version tags (+cu128) are stripped; the jax pins of the old .venv are dropped because pyroki needs jax < 0.4.30.
  • sam3 / graspnet pin setuptools<80; graspnet builds contact_graspnet without build isolation (its setup.py imports numpy).

Proxy: PyPI and the conda mirror go through http_proxy; GitHub (pyroki, lerobot, dlimp git dependencies) is fetched directly, because the proxy resets long git TLS connections.

Rebuilding: setup.sh reuses an existing env and only re-runs the post-install; delete <root>/envs/<name> to rebuild from scratch. It refuses to touch an env it did not create (marker file .harnesspai-env); an env whose create was interrupted before the marker was written is adopted if its micromamba history shows it was created from the same spec.

harnesspai env check on this machine (2026-10-21) — every env except robocasa passes:

[PASS] core        [PASS] libero     [PASS] robosuite   [PASS] sam3
[PASS] molmo       [PASS] pi05       [PASS] graspnet    [FAIL] robocasa   (not built)

Beyond the import check: libero resets a LIBERO-PRO environment, robosuite resets Lift, launch_sam3_server.py and launch_contact_graspnet_server.py import in their envs, molmo reports vllm 0.15 + CUDA, pi05 reports openpi + JAX on GPU with the pi05_libero checkpoint cached, and the test suite passes in core. The servers themselves have not been started (that occupies GPUs).

Check

micromamba run -p <root>/envs/core harnesspai env list     # which envs exist
micromamba run -p <root>/envs/core harnesspai env check    # import-check every env

configs/envs.yaml lists, per env, the modules env check imports.

LIBERO paths come from configs/libero/config.yaml via LIBERO_CONFIG_PATH (written by libero.post.sh); ~/.libero is not used.

3. Model services

Each service runs in its own env, one instance per GPU, port = base + 100 × gpu (configs/services.yaml). Logs and pid files go to logs/services/<name>/.

Service Env Port (gpu 0) Health
sam3 sam3 8114 /docs
molmo molmo 8122 /v1/models
pi05 pi05 8000 (single instance) /healthz
graspnet graspnet 8115 /docs
harnesspai serve up sam3 molmo --gpus 0,2 --wait
harnesspai serve up pi05 --wait
harnesspai serve status
harnesspai serve down sam3 --gpus 0,2

Scripts (auto GPU selection)

scripts/start_*.sh query every visible GPU with nvidia-smi, sort by free memory then utilization, and start one instance per selected GPU:

scripts/start_molmo.sh                 # Molmo2/vLLM on the 7 idlest GPUs
scripts/start_sam.sh                   # SAM3 on the 7 idlest GPUs
scripts/start_pyroki.sh                # PyRoKi IK server on one free GPU
scripts/start_pi05.sh                  # pi0.5 policy server on one free GPU

Arguments and overrides:

How Example
explicit GPUs (positional) scripts/start_sam.sh 0,2,3
env GPU_LIST=0,2 scripts/start_sam.sh
GPs count / wait / free-memory floor N=4 WAIT=0 MIN_FREE_MIB=60000 scripts/start_pi05.sh

If the host exposes fewer GPUs than requested the scripts warn and use all of them (--strict on the GPU picker turns that into an error). Indices are mapped back through CUDA_VISIBLE_DEVICES when it is set. Bare python -m harnesspai.infra.gpu lists the GPUs with free memory and utilization.

GPU_LIST also matters for consistency: a rollout on GPU g talks to the instance on GPU g through SAM3_SERVICE_URL / MOLMO_SERVICE_URL (base + 100 × g), so start at least as many service instances as rollout GPUs.

Models, gateway URL and proxy bypass live in configs/models.yaml. The Gemini diagnoser needs a key: vlm.api_keys in configs/local.yaml, or API_KEY (several comma-separated in API_KEYS for rotation). Codex uses coder.api_key in configs/local.yaml, else ~/.codex/config.toml.

4. Run

All commands run in the core env (except graph, assess and perceive, see Steps 1-3 below); rollouts are launched in the benchmark env named by configs/benchmarks/<bench>.yaml.

Language of terminal output, --help and the human-readable reports (REPORT.md, phase_report.md, perception README.md): harnesspai --lang en|zh <cmd> ..., or HARNESSPAI_LANG=en|zh, or lang: en|zh in configs/local.yaml (default zh). Prompts and the run artifacts agents read back (transcript, trace, round_report.md, ...) are always English.

# optional step 1: expert video -> workflow graph / graph-guided expert notes
harnesspai graph  --suite libero_object --task-id 0
harnesspai expert --suite libero_object --task-id 0
# optional step 2: VLA capability (rollouts, then per-phase good/needs_code/mixed)
harnesspai assess run     --benchmark libero_object
harnesspai assess perturb --benchmark libero_object
harnesspai assess phase   --eval-dir runs/libero/libero_object/1_capability --graph-dir harnesspai/memory/graph/data/libero/libero_object
# optional step 3: perception module
harnesspai perceive --bench libero_object --task-id 0
# step 4: main loop
harnesspai evolve "pick up the alphabet soup and place it in the basket" \
    --target <task package>/planning.py --task-module <module> \
    --graph harnesspai/memory/graph/data/libero/libero_object/task0.json \
    --suite libero_object --task-pkg task0 --task-id 0 \
    --eval-suites libero_object,libero_object_swap,libero_object_task \
    --episode-start 50 --episode-end 65 [--dry-run]

Task packages live only in harnesspai/code_library/ (scaffolded, evolved and promoted there); nothing is resolved from cap-x.

Step 1: build a workflow graph from the expert demo

Runs in the libero env (rendering the expert *_demo.hdf5 needs libero + h5py + imageio). Gemini watches the expert video and returns a phase list; Codex turns it plus a few keyframes into the graph JSON, which is validated and written to the graph library.

cd <repo>
export API_KEY=sk-...        # Gemini key (or vlm.api_keys in configs/local.yaml)
HarnessPAI=<mamba_root>          # configs/envs.yaml

# one task
micromamba run -p $HarnessPAI/envs/libero harnesspai graph --suite libero_goal --task-id 0
# whole suite (task0-9) + graphs.md
micromamba run -p $HarnessPAI/envs/libero harnesspai graph --suite libero_goal
# any mp4 (non-LIBERO)
micromamba run -p $HarnessPAI/envs/libero harnesspai graph --video /path/expert.mp4 \
    --task "lift the cube" --benchmark robosuite --suite lift --task-id 0
Output Path
graph harnesspai/memory/graph/data/<bench>/<suite>/taskN.json (e.g. libero/libero_goal/task0.json)
mermaid overview (whole suite) harnesspai/memory/graph/data/<bench>/<suite>/graphs.md
intermediates (expert mp4, keyframes, Gemini phases, Codex workdir) runs/_work/graph/<bench>/taskN/

Options: --view agentview|eye_in_hand, --max-frames 40 (frames sampled for Gemini), --n-keyframes 6 (keyframes for Codex), --model (override the Codex model), --out <path> (write elsewhere instead of the library), --quiet.

Re-running overwrites the library's taskN.json; the graphs are tracked in git, so git checkout restores them. Use --out /tmp/task0.json to compare first.

Step 2: assess the VLA's capability per graph phase

Runs in the libero env with the VLA server up. It measures where the VLA (default π0.5) succeeds or fails on each graph action phase, so the main loop only asks Codex to write code for the phases the VLA cannot do. Uses the graphs from Step 1.

cd <repo>
export API_KEY=sk-...        # Gemini key (phase tracing)
HarnessPAI=<mamba_root>          # configs/envs.yaml
SUITE=libero_goal

# 0. VLA server (port 8000)
micromamba run -p $HarnessPAI/envs/core harnesspai serve up pi05 --wait

# 1. base suite: classify -> rollout -> aggregate (skips stages already done)
micromamba run -p $HarnessPAI/envs/libero harnesspai assess run --benchmark $SUITE

# 2. LIBERO-PRO perturbations (swap / task), same episode window as the base
#    (needs the 100-state .init; generate it once if missing:
#     python -m harnesspai.envs.libero.init_states --suite ${SUITE}_swap)
micromamba run -p $HarnessPAI/envs/libero harnesspai assess perturb --benchmark $SUITE

# 3. per-phase verdict: good / needs_code / mixed
micromamba run -p $HarnessPAI/envs/libero harnesspai assess phase \
    --eval-dir runs/libero/$SUITE/1_capability \
    --graph-dir harnesspai/memory/graph/data/libero/$SUITE
Sub-step What it does Output (under runs/libero/<suite>/1_capability/)
classify (in run) Codex labels every graph action node keep / drop harnesspai/evolution/stages/capability/node_policy_<bench>_<suite>.json
rollout (in run) VLA rollouts on the base suite base_test/taskN/results.jsonl, base_test/taskN/epM/{run.log, success.mp4|failure.mp4}
aggregate (in run) success rates + graph split _summary.json
perturb the same rollouts on <suite>_swap / <suite>_task swap_test/, task_test/ (each with _summary.json, REPORT.md)
phase Gemini traces success/failure videos onto graph phases phase_capability/phase_summary.json, phase_report.md

evolve --only-needs-code-phases reads phase_summary.json and gives Codex only the needs_code phases; good / mixed phases are delegated to the VLA.

Common options:

  • run: --tasks 0-9, --ep-start 50 --ep-end 70 (exclusive), --max-parallel 5, --policy pi05, --port 8000, --only classify|rollout|aggregate, --force (or --force-classify / --force-rollout / --force-aggregate), --no-video.
  • perturb: --perturbations swap,task, --tasks, --ep-start 50 --ep-end 65, --force.
  • phase: --task-id N (repeatable), --n-success 3 --n-failure 3, --max-frames 48, --mode codex|direct, --force.

Step 3: evolve the perception module

Runs in the libero env with the SAM3 + Molmo fleet up (one pair per GPU in fleet_gpus, configs/services.yaml). It produces, per task, a verified perceive_taskN.py that localizes every instruction noun / relation with SAM3 + Molmo only (no VLM at runtime). Step 4 calls this module as-is instead of writing perception code. Reads the Step 2 rollout videos (1_capability/{base,swap,task}_test) and the task's task_prompts.json in Task Memory.

cd <repo>
export API_KEY=sk-...        # Gemini key (ready-frame pick, verification)
HarnessPAI=<mamba_root>          # configs/envs.yaml
SUITE=libero_object

# 0. perception fleet: SAM3 + Molmo on every fleet GPU
scripts/start_sam.sh 0,2,3,4,5,6,7
scripts/start_molmo.sh 0,2,3,4,5,6,7

# 1. (recommended) gold-standard scene labels: one labeled.png per task, used by the
#    verifier as a cross-check. Run on the whole suite so round 2 (perturbed nouns)
#    can borrow cross-task references.
micromamba run -p $HarnessPAI/envs/libero \
    python -m harnesspai.evolution.stages.perception.scene_label --suite $SUITE

# 2. perception evolution (base + swap + task episodes jointly)
micromamba run -p $HarnessPAI/envs/libero harnesspai perceive --bench $SUITE --task-id 0
micromamba run -p $HarnessPAI/envs/libero harnesspai perceive --bench $SUITE --tasks 0-9

Per task:

  1. parse the instruction into {nouns, relations} and seed prompts.json;
  2. for every episode in [ep-start, ep-end) Gemini picks the "ready" frame from the first --n-candidates frames of the rollout mp4;
  3. loop for at most --max-rounds rounds: run perceive_taskN.py on every ready frame (episodes spread over the fleet GPUs) → Gemini judges every noun / relation pass/fail on the overlay → if any item fails, Codex edits perceive_taskN.py (syntax-checked, rolled back on error) from the failures;
  4. converged = every item passes on every episode of every variant; Codex then generalizes the code into a skill and a per-task README.md is written.
Output Path
scene labels runs/<bench>/<suite>/2_perception/scene_label/taskN/{labeled,ready_frame,candidates}.png
perception code (main artifact) runs/<bench>/<suite>/2_perception/taskN/perceive_taskN.py
query phrasings / notes .../taskN/prompts.json, .../taskN/notes.md (optional, hand-written)
per-round log and verdicts .../taskN/evolution.log, round{N}.json
per-episode result + overlay .../taskN/perception/, ready frames in .../taskN/ready_frames/
convergence marker .../taskN/converged.json (a converged task is skipped on re-run)
suite report runs/<bench>/<suite>/2_perception/REPORT.md
generalized skill harnesspai/memory/skill/data/perception/taskN_strategy.py

Options: --bench libero_10|libero_goal|libero_object|libero_spatial (_swap / _task suffix with --no-joint runs one variant only, output under 2_perception/{swap,task}/), --tasks 0-9 / --task-id N, --ep-start 50 --ep-end 66 (exclusive), --max-rounds 6, --n-candidates 10, --max-workers N (use only the first N fleet GPUs), --codex-model, --rollouts-dir, --out-dir, --force (ignore converged.json and reseed the prompts).

Step 4: evolve the task code (main loop)

Runs in the core env; every rollout is launched in the benchmark env from configs/benchmarks/<bench>.yaml. Needs the VLA server (π0.5) and, if the task code uses perception, the SAM3 + Molmo services on the rollout GPUs. Steps 1-3 are optional inputs: the graph (Step 1) is required, phase_summary.json (Step 2) is used with --only-needs-code-phases, and 2_perception/taskN/perceive_taskN.py (Step 3) is injected automatically when it exists (else prompts.json wording).

Each round works on an isolated candidate copy; the code library is changed only after a full successful batch:

Step What
① Write Codex edits the target files (round 0 from the task, later rounds from the ③④ instruction)
② preflight python -m <task module> --help + task identity; round 0 adds a one-episode probe; failures go back to the writer in the same round (--smoke-retries)
①.6 Probe re-run up to 3 episodes that failed last round and fix until they pass (--probe-max-attempts)
② Execute open-loop rollout of every episode × every suite in --eval-suites
②.5 Diagnose Gemini watches the rollouts + graph against a baseline video (base suite: --expert-hdf5; swap/task: a success.mp4 from Step 2)
③④ Synthesize Codex read-only self-review → next round's instruction

Converged = every episode of every suite succeeds (a reviewer saying CONVERGED does not stop the loop). The candidate is then promoted into harnesspai/code_library/<bench>/<suite>/<task_pkg>/ and the run's failure→fix pairs are written to the skill library (--no-skill-writeback disables it). At least 5 rounds are budgeted (--rounds, default 10).

cd <repo>
export API_KEY=sk-...        # Gemini (②.5)
HarnessPAI=<mamba_root>          # configs/envs.yaml
PKG=harnesspai/code_library/libero/object/task0

micromamba run -p $HarnessPAI/envs/core harnesspai serve up pi05 --wait

# check the plan and the run dir first (no agents, no rollout)
micromamba run -p $HarnessPAI/envs/core harnesspai evolve \
    "pick up the alphabet soup and place it in the basket" \
    --target $PKG/planning.py --task-module harnesspai.code_library.libero.object.task0 \
    --graph harnesspai/memory/graph/data/libero/libero_object/task0.json \
    --suite libero_object --task-pkg task0 --dry-run

# real run: base + swap + task, episodes 50..64, 4 rollout GPUs
micromamba run -p $HarnessPAI/envs/core harnesspai evolve \
    "pick up the alphabet soup and place it in the basket" \
    --target $PKG/planning.py --task-module harnesspai.code_library.libero.object.task0 \
    --graph harnesspai/memory/graph/data/libero/libero_object/task0.json \
    --suite libero_object --task-pkg task0 \
    --eval-suites libero_object,libero_object_swap,libero_object_task \
    --episode-start 50 --episode-end 65 --eval-gpus 0,2,3,4 \
    --expert-hdf5 <LIBERO-PRO demo .hdf5> --only-needs-code-phases

Missing target files are scaffolded in the candidate (no sibling task code is copied); --target is repeatable. --task-id defaults to the number in --task-pkg.

Output (under runs/<bench>/<suite>/3_evolution/<task_pkg>__<timestamp>/) Contents
transcript.md the run's story line
run_meta.json, success_contract.md metadata; the success contract every round is judged against
candidate/ the isolated working copy Codex edits
runs/round_NNN/codex_edit.md, diff.md, code/ what Codex wrote this round
runs/round_NNN/rollout*/, probe_*/, trace.md, stderr.log rollout and probe results
runs/round_NNN/video_diagnosis.md, codex_analysis.md, round_report.md ②.5 diagnosis, ③④ review, human-readable round summary
final/diff.md final diff against the starting targets

Common options: --rounds 10, --eval-parallel N (episodes per GPU), --resume-from <run dir> (continue an unfinished run's candidate in a new run), --codex-model, --codex-timeout 1800 / --codex-synth-timeout 1800, --no-video-diagnosis, --vdm-model / --vdm-url / --vdm-key, --vla-eval-dir (default 1_capability/phase_capability), --benchmark (default: inferred from the --graph path), --run-dir, --no-verbose.

Step 5: batch-evaluate a promoted package

harnesspai evolve converges on one episode window; this runs the same package over a wider window, with every selected GPU carrying several simulators at once (one episode = one process, so --per-gpu 5 really means five LIBERO sims per GPU). Workers on GPU g talk to the SAM3/Molmo instance on GPU g (base + 100·g, see §3), so start as many service instances as rollout GPUs.

# 7 idlest GPUs × 5 envs = 35 concurrent sims, base + swap + task suites
scripts/run_eval_libero.sh libero_object task0

# explicit GPUs, wider window, base suite only
SUITES=libero_object EPISODES=0-100 PER_GPU=5 \
    scripts/run_eval_libero.sh libero_object task0 0,1,2,3

# evaluate a package that has not been migrated into harnesspai/code_library yet
PACKAGE=third_party/cap-x/capx/eva/code_library/libero/object/task0 \
    scripts/run_eval_libero.sh libero_object task0 0,1

The script picks GPUs, then calls python -m harnesspai.execution.evaluate. Before fanning out it runs one real episode as a runtime probe — --help only reaches argparse, so an import that fails inside run_task would otherwise crash every worker and leave N empty episode dirs.

Output (under runs/<bench>/<suite>/4_eval/<task_pkg>__<ts>/) Contents
plan.json module, package dir, python, suites, episodes, GPUs, workers
summary.md, summary.json per-suite success count/rate, failed episodes, wall time
_probe/epNNN/ the probe episode (stdout/stderr/exit code)
<suite>/epNNN/ *.mp4, diagnostics.jsonl, phase_frames.json, stdout.log, stderr.log, exit_code

Success is read from diagnostics.jsonl alone. Exit codes are not a signal: the task packages disagree (libero goal/object return 1 on success, robosuite and robocasa return 0, libero/spatial returns a success count), so a successful episode would otherwise read as a failure.

📄 Paper & Citation

HarnessPAI: An Evolving Harness for Physical AI
Darwin Agent Team

Read the paper · Explore the project

@article{wang2026harnesspai,
  title   = {{HarnessPAI}: An Evolving Harness for Physical AI},
  author  = {Xin Wang and Wenhao Wu and Menghao Zhang and Zhi Wang and Kun Shao and Jian Luan and Yang Li and Qing Li and Shangding Gu and Huichi Zhou and Shuqing Shi and Fei Ni and Shuo Lu and Weicheng Meng and Kang Li and Jin Wu and Kang Zhao and Shangmin Guo and Gen Li and Yongqiang Tang and Zhizhong Zhang and Yuan Xie and Heng Qu},
  journal = {arXiv preprint arXiv:2609.29166},
  year    = {2026}
}
HarnessPAI — Execute programs. Learn from outcomes. Evolve the harness and models.
By the Darwin Agent Team

About

No description, website, or topics provided.

Resources

Stars

51 stars

Watchers

2 watching

Forks

Releases

Packages

Contributors

Languages