A benchmark for household robot agents, from individual manipulation tasks to open-ended assistance.
Website · Simulation assets · InternVLA model · Quick start · Documentation
overview.mp4
RobotAgentBench is a simulation benchmark for household robot agents. Using the AgiBot G1 embodiment, agents observe the home, identify what needs to be done, plan their actions, and execute them through a shared observation/action interface.
The benchmark spans everyday manipulation, task discovery, context reasoning, failure recovery, robustness, and open-ended scenarios. Task-specific process scores measure verified physical progress, including partial completion.
This repository contains the task backends, initialization and randomization, scoring, and evaluation runtime. Versioned simulation assets are available separately on Hugging Face.
| Evaluation | Scope | What it measures |
|---|---|---|
| Manipulation | 18 short tasks and 8 long tasks | Executing explicit household instructions, from individual manipulations to activities with multiple subgoals |
| Task Discovery | Scenario-conditioned exploration | Identifying and touching task-relevant objects |
| Context Reasoning | Drink, snack, waste, and sofa-object selection | Choosing the correct object from context |
| Failure Recovery | Predefined failed-action states | Recognizing a failed state and adapting subsequent actions |
| Robustness | Object movement and lighting disturbances | Continuing the objective as execution conditions change |
| OpenEnd | Dinner, guests, leaving, and tidying | Inferring, planning, and executing tasks from a high-level request |
Manipulation includes both short and long tasks. The reported Manipulation score uses the 18 short tasks; the 8 long tasks are evaluated separately.
See the benchmark guide for the task list and scoring distinctions.
Use Python 3.11 or newer and check out the evaluation code's main branch:
git clone --branch main https://github.com/AetherLabsAI/RobotAgentBench.git
cd RobotAgentBench
python3 -m venv .venv
. .venv/bin/activate
python -m pip install .
python -m robotagentbench --help
robotagentbench catalog standaloneThe core package and catalog inspection do not require Isaac Sim, a GPU, or model credentials. Native simulation uses a separate environment described below.
The asset release 0.1.7 is approximately 7.04 GB, split into 33 archives, including the license archive. It contains all 50 original room layouts and their shared runtime dependencies, robot resources, task objects, initialization data, and the full Context Reasoning question banks and standard answers. Use it with configuration bank 0.2.17. All 32 earlier archives are reused unchanged; the new shared scene-reconstruction supplement is approximately 516 KB. Update the evaluator and install into fresh asset and runtime directories when upgrading.
robotagentbench assets fetch resources/assets-0.1.7.json --cache downloads
robotagentbench assets install resources/assets-0.1.7.json --cache downloads --destination Assets
robotagentbench assets configure resources/assets-0.1.7.json --destination Assets --runtime Runtime --profile roomThe manifest pins an immutable asset revision and verifies file hashes. The commands create fresh Assets and Runtime directories; existing installations are not overwritten. For offline setup, place all archive parts in downloads and skip fetch. See the asset guide for details.
Standard answers are evaluator data and are not included in agent observations.
Native execution requires Linux x86_64, Docker, NVIDIA Container Toolkit, and an Isaac Sim-compatible NVIDIA GPU. Build the pinned native environment, then prepare the coffee task:
docker build -f docker/Dockerfile -t robotagentbench:native .
python -m robotagentbench.namespace --runtime Runtime --image robotagentbench:native --plan
python -m robotagentbench.namespace --runtime Runtime --image robotagentbench:native --prepare-native --name coffee_01
python -m robotagentbench.namespace --runtime Runtime --image robotagentbench:native --prepare --name coffee_01--prepare-native creates this episode's private Kit/MDL caches and NVIDIA ICD configuration inside a CPU-only container. No host SDK installation is needed. Use the same image for preparation and execution; a changed image or an existing deployment is rejected.
Review the NVIDIA license setup. If you accept the applicable terms, start the task worker:
export OMNI_KIT_ACCEPT_EULA=YES
mkdir -p gpu-locks
python -m robotagentbench.namespace --runtime Runtime --image robotagentbench:native \
--run --name coffee_01 --gpu 0 --lock-root gpu-locksThe worker exposes a JSONL observation/action interface for an agent or recorded action source. Starting the worker does not run a policy automatically. Follow the coffee walkthrough for interaction examples and the runtime guide for agent integration. Use the same GPU-lock directory for all jobs on a host.
The saved-instance release provides 30 pre-generated initial configurations per task or scenario: 540 for the 18 short tasks, 240 for the 8 long tasks, 570 across the four additional evaluation dimensions, and 120 for the four OpenEnd scenarios (1,470 in total). Choose an episode from 1–30 and reuse its saved initial state; running a model does not resample it.
Configuration bank 0.2.17 includes Today's situation in all 120 OpenEnd configurations. Each scenario combines six context/scene variants with five spatial initial states. Context supplies circumstances and preferences; stains, waste, object locations, and cleanliness must be discovered from camera observations.
This release adds six original-rule replacements whose initial states passed
their task-owned native admission checks. The other 1,464 states are
byte-identical to 0.2.16; public contexts, scoring, physics, shared assets 0.1.7,
and all 50 room layouts are unchanged. All 1,470 saved initial configurations
have passing original initial-admission records; this does not claim task
completion or model success. Use evaluator commit e8448f732572cacf330554c3620d6f25341be81f
or later. See configuration details and release history.
The configuration download is approximately 260 MB in seven archives, separate from the shared assets. Room bases are stored once; configurations contain compact lossless state records and patches rather than complete room copies. Cloth tasks retain their required initial-state arrays.
These are release configurations for reproducible evaluation, not the exact instances used for the paper's reported results. Episode numbers identify saved configurations, not consecutive sampling seeds; the bank records each configuration's actual seed.
See saved configurations for download, selection, and read-only execution setup.
Two optional clients reuse the project's server evaluation code:
- InternVLA-A1.5: download our fine-tuned model from AetherLabs-AI/InternVLA-A1.5-RAB, then use the matching G1 statistics and action projection.
- Codex Robo Agent: the App Server runner and robot tools, with a Dockerfile and locked dependencies for a locally built code-execution sandbox. No server-specific image ID is required.
Start with shared model setup, then follow either
client's complete restore → configure → run walkthrough. Both reuse the
Docker native image; no host Isaac SDK paths or hand-written worker JSON are
needed. The examples select prepare_coffee, saved episode 1.
After following the configuration step in the selected guide:
# InternVLA: separate native/model GPUs; RAB_ROOT and RAB_PY come from shared setup.
"$RAB_PY" -m robotagentbench.model_examples.docker_workflow run \
"$RAB_ROOT/model-configs/internvla-coffee01/run.json" --execute
# Codex: uses your own Codex login and can consume model usage.
"$RAB_PY" -m robotagentbench.model_examples.docker_workflow run \
"$RAB_ROOT/model-configs/codex-coffee01/run.json" --executeOmit --execute to inspect without starting either model or simulator. The
generic coffee worker in step 3 is not a model run; the client walkthroughs
prepare their own fresh native caches and run configurations. Dedicated cloth
and specialized task owners keep their separate interfaces. Neither client
bundles credentials or changes task scoring.
Agents receive synchronized camera observations, measured robot-state feedback, task-provided context, and an explored navigation map through the generic native worker. The map shows known static geometry, robot pose, and the executed path—not task targets or planned routes. Agents submit physical action chunks; the standard room profile uses one head camera and two wrist cameras with a 20 Hz control interface. Each task handles its own process scores, completion conditions, and termination. See the observation contract for field names and execution-profile scope.
Results include process scores, completion status, and a termination reason. Setup and runtime errors are reported separately from scored episodes. Camera, physics, and execution settings for specialized tasks are described in the task backend guides.
| Guide | Start here to… |
|---|---|
| Benchmark | Understand task families, scenarios, and evaluation scope |
| Installation | Install the evaluator and native dependencies |
| Coffee example | Install assets, prepare an episode, and connect to the task worker |
| Task backends | Locate task implementations and execution profiles |
| Assets | Download, install, and verify a versioned asset release |
| Saved configurations | Select one of 30 saved initial configurations without resampling |
| Shared model setup | Prepare Docker, assets and saved inputs for either model client |
| InternVLA example | Download the fine-tuned model and run a VLA policy |
| Codex example | Configure and run a code-based Robo Agent |
| CLI reference | Inspect catalogs, verify resources, and invoke configured native operators |
| Tablecloth tasks | Configure the separate cloth environment |
| Recovery and coffee variants | Configure specialized execution environments |
| Runtime | Integrate workers, observations, actions, scoring, and shutdown |
| Licenses and notices | Read asset and dependency licensing terms |
src/robotagentbench/
├── tasks/ Task and OpenEnd component catalogs
├── backends/ Task implementations and versioned source profiles
├── interfaces/ Robot state and action contracts
├── runtime/ Action transport and session adapters
└── simulation/ Source admission, native workers, and resource lifecycle
docs/ Setup, benchmark, integration, and resource guides
tests/ CPU contract and lifecycle tests
For questions and bug reports, use GitHub Issues. Include the task identifier, code revision, execution profile, and a minimal reproduction. Do not include credentials or sensitive configuration.
Run the CPU tests from the repository root:
python -m unittest discover -s tests- Evaluation code: Apache-2.0.
- Team-authored assets and task data: CC BY 4.0.
- Third-party materials: retain their original terms, including CC BY-NC-SA 4.0 for the AgiBot GenieSim robot resources. See third-party notices and the asset notices.
If you use RobotAgentBench in your research, please cite:
@misc{mao2026robotagentbenchopenworldrobotautonomy,
title={RobotAgentBench: Towards Open-world Robot Autonomy},
author={Lingjun Mao and Jinglin Cao and Lukun He and Wenpeng Xu and
Yuchen Yan and Hengyu Jin and Zijun Zhang and Jinzhou Tang and
Songyao Jin and Ziche Liu and Yifei Shao and Junbo Huang and
Fang Nan and Ruobing Han and Ziqiao Xi and Ziming Xu and
Shuang Liang and Sibo Zhu and Wenyi Wu and Xinyue Wang and
Yiming Huang and Xinkai Zou and Gao Zhu and Kun Zhou and Biwei Huang},
year={2026},
eprint={2610.34649},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2610.34649},
}Scan a code to join the RobotAgentBench group on WeChat, Slack or Discord, or open an invite link directly.
 |
 Slack |
 Discord |