ViMoS — Do You Still Need Mocap?

ViMoS is an end-to-end pipeline for humanoid robot motion imitation — from raw video to deployed policy on a real robot. It chains four stages: video → SMPL pose (GENMO) → robot motion (GMR) → policy training (BeyondMimic) → deploy.

Supports Booster T1 and Unitree G1 (and 15+ other humanoids via GMR).

Pipeline Overview

Video (.mp4)
    │
    ▼
[Stage 1] GENMO — Video → SMPL-X pose estimation
    │  retarget/GENMO/
    │
    ▼
[Stage 2] GMR — SMPL-X → Robot joint motion (CSV/PKL)
    │  retarget/GMR/
    │
    ▼
[Stage 3] whole_body_tracking — CSV → NPZ → Policy training (Isaac Lab / PPO)
    │  train/whole_body_tracking/
    │  exports: ONNX (G1) or TorchScript JIT (T1)
    │
    ▼
[Stage 4] Deploy — Policy inference on MuJoCo sim or real robot (ROS 2)
           deploy/motion_tracking_controller/   ← G1 (ONNX)
           deploy/booster_deploy/               ← T1 (JIT)

Repository Structure

ViMoS/
├── scripts/
│   ├── video_to_robot.py   ← main pipeline (venv)
│   └── video_to_robot_docker.py   ← pipeline via Docker
├── retarget/
│   ├── GENMO/                   ← video → SMPL-X (submodule)
│   └── GMR/                     ← SMPL-X → robot motion (submodule)
├── train/
│   └── whole_body_tracking/     ← BeyondMimic / Isaac Lab (submodule)
├── deploy/
│   ├── motion_tracking_controller/  ← ROS 2 inference — G1 (submodule)
│   └── booster_deploy/              ← T1 deploy (submodule)
└── docker-compose.yml

Setup

Clone with submodules

git clone --recurse-submodules https://github.com/AKCIT-RL/ViMoS.git
cd ViMoS

Or after cloning:

git submodule update --init --recursive

Download checkpoints & body models

The pipeline needs several model files. Some are downloaded automatically; a few are license-gated and must be downloaded once by hand (free registration).

Step 1 — download the license-gated files and drop them in the repo root:

File	Source (register + accept license)
`s050000.ckpt`	GENMO motion model — Google Drive
`models_smplx_v1_1.zip`	SMPL-X body models — smpl-x.is.tue.mpg.de
`SMPL_python_v.1.1.0.zip`	SMPL body model — smpl.is.tue.mpg.de

Just leave the three files in the repository root (ViMoS/) — no need to rename or unzip them.

Step 2 — run the installer:

python scripts/download_assets.py

This extracts and places every file in the exact location each stage expects, and automatically downloads the freely-redistributable GENMO video backbone checkpoints (GVHMR, ViTPose, HMR2) from a public mirror. It is idempotent — re-running it only fetches what is missing. If your archives are elsewhere, use --genmo-ckpt, --smplx-zip, and --smpl-zip; use --skip-download to avoid network access.

After it finishes, the resulting layout is:

retarget/GENMO/inputs/checkpoints/
├── s050000.ckpt                         # GENMO motion model
├── vitpose/vitpose-h-multi-coco.pth     # auto-downloaded
├── gvhmr/gvhmr_siga24_release.ckpt      # auto-downloaded
├── hmr2/epoch=10-step=25000.ckpt        # auto-downloaded
├── yolo/yolov8x.pt                      # auto-downloaded (person detector)
└── body_models/
    ├── smplx/SMPLX_NEUTRAL.npz          # (+ MALE/FEMALE)
    ├── smpl/SMPL_NEUTRAL.pkl
    ├── smplx2smpl_sparse.pt             # copied from the GVHMR submodule
    └── smpl_neutral_J_regressor.pt      # copied from the GVHMR submodule

retarget/GMR/assets/body_models/
└── smplx/SMPLX_NEUTRAL.npz              # (+ MALE/FEMALE)

The robot assets (URDF/meshes for every supported humanoid) ship inside the GMR submodule under retarget/GMR/assets/<robot>/ — no extra download needed.

On its first run, GENMO downloads the t5-3b text encoder (~11 GB) into retarget/GENMO/hf_cache/ (mounted as a volume). The download happens once and is reused on subsequent runs — make sure you have the free disk space.

Environment setup

Each stage has its own environment. You have two options:

Option A — Manual setup (per submodule README):

Stage	Submodule	Setup guide
Retarget (GENMO)	`retarget/GENMO/`	GENMO README — uv venv (`uv venv --python 3.10`)
Retarget (GMR)	`retarget/GMR/`	GMR README — uv venv (`uv venv --python 3.10`)
Training	`train/whole_body_tracking/`	WBT README — conda Isaac Lab 2.1 / Isaac Sim 4.5
Deploy G1	`deploy/motion_tracking_controller/`	MTC README — ROS 2 Jazzy
Deploy T1	`deploy/booster_deploy/`	Booster Deploy README

Option B — Docker (recommended for Stages 1–3):

Prebuilt images are published to GitHub Container Registry — pull them instead of building locally:

docker compose pull genmo gmr     # fetch prebuilt GENMO + GMR images

Or build from source:

docker compose build              # GENMO + GMR
docker compose --profile wbt build   # + whole_body_tracking (Isaac Lab)
docker compose --profile mtc build  # + motion_tracking_controller (ROS 2)

Requires Docker Engine 20.10+ and nvidia-container-toolkit. Run python scripts/download_assets.py first so the checkpoints are in place (see Download checkpoints & body models). The wbt image builds on Isaac Sim and requires docker login nvcr.io (free NGC account) before its first build.

Stage 1 + 2 — Video → Robot Motion

With Docker (recommended)

# Single video
python scripts/video_to_robot_docker.py --video /path/to/video.mp4 --robot booster_t1

# Folder of videos
python scripts/video_to_robot_docker.py --videos_path /path/to/folder/ --robot unitree_g1

# With visualization video and headless mode (no display, for SSH)
python scripts/video_to_robot_docker.py --video video.mp4 --robot booster_t1 --record_video --headless

With venv (manual)

python scripts/video_to_robot.py --video /path/to/video.mp4 --robot booster_t1

Activate the GENMO venv (retarget/GENMO/.venv) before running. The script manages both internally via GENMO_PYTHON and GMR_PYTHON.

Sandwich mode (default: disabled)

Some videos start or end abruptly — mid-motion, mid-kick, or in an unstable pose. When the robot trains on this reference as-is, it may learn to not maintain balance in the final pose, since the reference itself never shows a stable recovery.

Sandwich mode wraps the motion with "stand still" anchor frames at both ends:

[30 frames stand still] + [motion from video] + [60 frames stand still]

This gives the policy a clear signal for what a safe start and end look like.

Use it when: the video starts/ends in a pose the robot cannot safely hold (e.g. mid-kick, leaning, one foot up).
Skip it when: the video already starts and ends with the robot standing still.

To enable:

python scripts/video_to_robot.py --video video.mp4 --robot booster_t1 --sandwich

Outputs

File	Description
`retarget/GENMO/outputs/demo/<name>/hmr4d_results.pt`	SMPL-X pose estimation
`retarget/GMR/output/pkl/<robot>_<name>.pkl`	Retargeted robot motion
`retarget/GMR/output/csv/<robot>_<name>.csv`	Robot motion as CSV (joints × frames)
`retarget/GMR/videos/<robot>_<name>.mp4`	Visualization (with `--record_video`)

Supported robots

booster_t1   booster_t1_29dof   booster_k1
unitree_g1   unitree_g1_with_hands   unitree_h1   unitree_h1_2
fourier_n1   engineai_pm01   kuavo_s45   hightorque_hi
galaxea_r1pro   stanford_toddy   berkeley_humanoid_lite
pnd_adam_lite   openloong   tienkung

Stage 3 — Policy Training

Uses BeyondMimic (paper) — PPO in Isaac Lab with adaptive sampling and anchor-frame motion tracking. The same reward formulation works for any motion without parameter tuning.

Prerequisites

NVIDIA GPU (RTX 3090+ recommended)
Isaac Lab 2.1 / Isaac Sim 4.5 (or Docker with NGC login)
W&B account for artifact registry

Step 1 — Convert CSV → NPZ and upload to W&B

cd train/whole_body_tracking

# Booster T1
python scripts/csv_to_npz.py \
  --input_file /path/to/motion.csv \
  --input_fps 30 \
  --output_name my_motion \
  --robot booster_t1 \
  --wandb_project Booster_t1 \
  --headless

# Unitree G1
python scripts/csv_to_npz.py \
  --input_file /path/to/motion.csv \
  --input_fps 30 \
  --output_name my_motion \
  --wandb_project G1_motions \
  --headless

Step 2 — Train

Booster T1:

python scripts/rsl_rl/train.py \
  --task=Tracking-Flat-T1-Wo-State-Estimation-v0 \
  --num_envs 4096 \
  --headless --video \
  --logger wandb \
  --log_project_name <project_name> \
  --registry_name <entity>/<project_name>/<motion_name>:latest \
  --experiment_name <motion_name> \
  --run_name <motion_name>

Unitree G1:

python scripts/rsl_rl/train.py \
  --task=Tracking-Flat-G1-v0 \
  --num_envs 4096 \
  --headless --video \
  --logger wandb \
  --log_project_name <project_name> \
  --registry_name <entity>/<project_name>/<motion_name>:latest \
  --experiment_name <motion_name> \
  --run_name <motion_name>

Resume training from checkpoint

python scripts/rsl_rl/train.py \
  --task=Tracking-Flat-T1-Wo-State-Estimation-v0 \
  --num_envs 4096 \
  --headless --video \
  --logger wandb \
  --log_project_name <project_name> \
  --registry_name <entity>/<project_name>/<motion_name>:latest \
  --experiment_name <motion_name> \
  --run_name <motion_name> \
  --resume 1 \
  --load_run <run_folder_name> \
  --checkpoint model_XXXXX.pt

Available tasks

Robot	Task	Notes
Booster T1	`Tracking-Flat-T1-Wo-State-Estimation-v0`	Use this — hardware has no state estimator
Unitree G1	`Tracking-Flat-G1-v0`	Full observations
Unitree G1	`Tracking-Flat-G1-Wo-State-Estimation-v0`	Without state estimation
Unitree G1	`Tracking-Flat-G1-Low-Freq-v0`	Low-frequency control

Step 3 — Export policy

# Booster T1 — exports TorchScript JIT (.pt)
python scripts/rsl_rl/play.py \
  --task=Tracking-Flat-T1-Wo-State-Estimation-v0 \
  --num_envs 1 \
  --headless \
  --wandb_path <entity>/<project_name>/<run_id>

# Unitree G1 — exports ONNX (.onnx)
python scripts/rsl_rl/play.py \
  --task=Tracking-Flat-G1-v0 \
  --num_envs 1 \
  --headless \
  --wandb_path <entity>/<project_name>/<run_id>

Exported files go to logs/rsl_rl/temp/exported/.

Batch training (T1)

export WANDB_ENTITY=<your-entity>
bash scripts/batch_csv_train.sh

Edit the motion list inside scripts/batch_csv_train.sh before running.

Stage 4 — Deploy

Booster T1 (TorchScript JIT)

See deploy/booster_deploy/README.md.

Place the exported .pt file as required by the deploy package and follow the startup instructions.

Unitree G1 (ONNX — ROS 2)

# MuJoCo simulation
ros2 launch motion_tracking_controller mujoco.launch.py \
  policy_path:=/path/to/policy.onnx

# Real robot (use with extreme care)
ros2 launch motion_tracking_controller real.launch.py \
  policy_path:=/path/to/policy.onnx

Via Docker:

xhost +local:docker
docker compose --profile mtc run --rm motion-tracking-controller \
  bash -c "ros2 launch motion_tracking_controller mujoco.launch.py policy_path:=/workspace/data/policy.onnx"

Safety: Running policies on a real robot is dangerous and for research only. See the disclaimer in deploy/motion_tracking_controller/README.md.

T1 vs G1 — Quick Reference

	Booster T1	Unitree G1
Retarget flag	`--robot booster_t1`	`--robot unitree_g1`
csv_to_npz flag	`--robot booster_t1`	(default)
Training task	`Tracking-Flat-T1-Wo-State-Estimation-v0`	`Tracking-Flat-G1-v0`
Export format	TorchScript JIT (`.pt`)	ONNX (`.onnx`)
Deploy	`deploy/booster_deploy/`	`deploy/motion_tracking_controller/`
Max iterations	15.000	30.000

Docker — All Services

Service	Profile	Description
`genmo`	(default)	Video → SMPL-X
`gmr`	(default)	SMPL-X → robot motion
`whole-body-tracking`	`wbt`	Isaac Lab training
`motion-tracking-controller`	`mtc`	ROS 2 G1 deploy

For wbt: run docker login nvcr.io before the first build (NGC API key required).

Troubleshooting

Empty submodules after clone: git submodule update --init --recursive
Update submodules: git submodule update --remote --checkout
Errors in each stage: check the corresponding submodule README
GENMO can't find a checkpoint: run python scripts/download_assets.py to (re)install every model file; see Download checkpoints & body models for the expected layout.
GMR no display (SSH): use --headless or Docker (MUJOCO_GL=egl set automatically)
W&B artifact not found: verify WANDB_ENTITY and the exact artifact name with wandb artifact ls <entity>/<project>

Vertical / phone-recorded videos

If your input videos were recorded vertically on a phone, they likely carry a rotate=90 (or 180/270) metadata tag while the actual pixels are stored in landscape orientation. The video reader used by GENMO/GVHMR (imageio + pyav) does not honor this rotation metadata, so the estimated human mesh will appear lying on the ground.

Re-encode such videos to bake the rotation into the pixels before running the pipeline:

mkdir -p videos_fixed
for f in videos/*.mp4; do
  ffmpeg -y -i "$f" \
    -c:v libx264 -crf 17 -preset fast \
    -c:a copy \
    -map_metadata 0 -metadata:s:v:0 rotate=0 \
    "videos_fixed/$(basename "$f")"
done

Then pass videos_fixed/ as input instead of the original folder.

Acknowledgements

ViMoS integrates:

GENMO — Human motion generation from video
GMR — General Motion Retargeting (Yanjie Ze et al.)
BeyondMimic / whole_body_tracking — Motion tracking policy training (HybridRobotics)
motion_tracking_controller — ROS 2 deployment
booster_deploy — Booster T1 deployment

Please refer to each submodule for credits, licenses, and upstream documentation.

License

See the LICENSE files in each submodule.

Navigation Menu

Repository files navigation

ViMoS — Do You Still Need Mocap?

Pipeline Overview

Repository Structure

Setup

Clone with submodules

Download checkpoints & body models

Environment setup

Stage 1 + 2 — Video → Robot Motion

With Docker (recommended)

With venv (manual)

Sandwich mode (default: disabled)

Outputs

Supported robots

Stage 3 — Policy Training

Prerequisites

Step 1 — Convert CSV → NPZ and upload to W&B

Step 2 — Train

Resume training from checkpoint

Available tasks

Step 3 — Export policy

Batch training (T1)

Stage 4 — Deploy

Booster T1 (TorchScript JIT)

Unitree G1 (ONNX — ROS 2)

T1 vs G1 — Quick Reference

Docker — All Services

Troubleshooting

Vertical / phone-recorded videos

Acknowledgements

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
deploy		deploy
docs		docs
retarget		retarget
scripts		scripts
train		train
.gitignore		.gitignore
.gitmodules		.gitmodules
README.md		README.md
docker-compose.yml		docker-compose.yml

Latest commit

History

Folders and files

Repository files navigation

ViMoS — Do You Still Need Mocap?

Pipeline Overview

Repository Structure

Setup

Clone with submodules

Download checkpoints & body models

Environment setup

Stage 1 + 2 — Video → Robot Motion

With Docker (recommended)

With venv (manual)

Sandwich mode (default: disabled)

Outputs

Supported robots

Stage 3 — Policy Training

Prerequisites

Step 1 — Convert CSV → NPZ and upload to W&B

Step 2 — Train

Resume training from checkpoint

Available tasks

Step 3 — Export policy

Batch training (T1)

Stage 4 — Deploy

Booster T1 (TorchScript JIT)

Unitree G1 (ONNX — ROS 2)

T1 vs G1 — Quick Reference

Docker — All Services

Troubleshooting

Vertical / phone-recorded videos

Acknowledgements

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages