Point2Pose — Synthetic Data Generator

This repository is the synthetic data generation code for Point2Pose (ECCV 2026) (project page), built on NVIDIA Isaac Lab / Isaac Sim.

It renders photorealistic RGB‑D observations of moving YCB objects from a virtual camera and logs, per frame, the object pose in the camera frame — everything needed to train and evaluate 6‑DoF pose estimation.

Three entry points are provided:

Script	Purpose
extensions/synthetic_data_generator/run_camera.py	Spawns YCB objects on a packing table, moves/rotates them along several trajectory types, and saves synchronized RGB, depth, camera intrinsics, and per-object camera-frame poses (plus an assembled video). This is the main data generator.
extensions/synthetic_data_generator/run_ycb_object_loader.py	Loads the full YCB object set on a ground plane for visualization / figures (press X to screenshot).
extensions/synthetic_data_generator/run_ycb_selected_shot.py	Loads a fixed set of 7 YCB objects in a row, renders them from one camera angle, and writes a single PNG to screenshots/ (headless one-shot).

---

Repository layout

Point2Pose-SyntheticDataGenerator/
├── extensions/
│   ├── __init__.py                       # defines POINT2POSE_SDG_ROOT_DIR (repo root)
│   ├── synthetic_data_generator/
│   │   ├── run_camera.py                 # main synthetic data generator
│   │   ├── run_ycb_object_loader.py      # YCB visualization loader
│   │   ├── run_ycb_selected_shot.py      # 7-object one-shot camera render
│   │   └── utils.py                      # camera params, logging, image/pose IO
│   ├── assets/grasp_object/              # GraspObject (RigidObject + bbox viz)
│   └── benchmarks/ycb_datasets.py        # YCB object lists, bbox dict, difficulty sets
├── IsaacLab/                             # git submodule: upstream Isaac Lab, pinned to v2.3.2
├── datasets/                             # YCB assets (git-ignored, see below)
│   ├── ycb/                              # google_16k meshes + converted USD + canonical transforms
│   └── YCB_Video_Models/                 # YCB-Video models + converted USD
├── requirements.txt
└── README.md

The top-level package is named extensions/ on purpose. The scripts import modules as extensions.synthetic_data_generator... / extensions.assets... / extensions.benchmarks..., and POINT2POSE_SDG_ROOT_DIR (in extensions/__init__.py) is derived from the repo-root location so that datasets/ resolves relative to the repo root. Run scripts from the repo root so extensions is importable.

Isaac Lab is vendored as the IsaacLab/ git submodule (upstream, pinned to v2.3.2) and installed editable into the conda env, so this repo is self-contained and reproducible — no external Isaac Lab checkout required.

---

1. Datasets (not tracked in git)

The YCB assets are large (~4 GB) and are git-ignored (datasets/ycb/, datasets/YCB_Video_Models/). The scripts consume pre-converted USD meshes and a precomputed ycb_canonical_transforms.npz — copy these provisioned artifacts from an existing checkout:

# ycb/  → google_16k meshes + per-object usd/ + ycb_canonical_transforms.npz
# YCB_Video_Models/  → YCB-Video models + per-object usd/
rsync -a --exclude='__pycache__' --exclude='meshes.zip' \
    /path/to/source/datasets/ycb \
    /path/to/source/datasets/YCB_Video_Models \
    datasets/

Expected per-object layout after provisioning:

datasets/ycb/<vNNN_name>/usd/<vNNN_name>.usd
datasets/ycb/ycb_canonical_transforms.npz
datasets/YCB_Video_Models/models/<NNN_name>/usd/<NNN_name>.usd

---

2. Environment setup

These scripts require Isaac Sim + Isaac Lab. Create a dedicated conda environment named point2pose_sdg:

# 0) Get the code + the Isaac Lab submodule
git clone --recurse-submodules <this-repo-url>
cd Point2Pose-SyntheticDataGenerator
# (existing clone: git submodule update --init --recursive)

# 1) Create the environment
conda create -n point2pose_sdg python=3.11 -y
conda activate point2pose_sdg

# 2) Install Isaac Sim 5.1 (pulls torch 2.7.0+cu126, numpy 1.26, and the CUDA stack)
pip install --upgrade pip
pip install "isaacsim[all,extscache]==5.1.0" --extra-index-url https://pypi.nvidia.com

# 3) Install Isaac Lab (editable) from the vendored submodule
pip install -e IsaacLab/source/isaaclab

# 4) Install the extra Python dependencies used by these scripts
pip install -r requirements.txt

Isaac Sim needs an NVIDIA GPU with a recent driver and CUDA-capable runtime. This code is developed against Isaac Sim 5.1.0 and Isaac Lab v2.3.2 (the pinned submodule).

Do not run IsaacLab/isaaclab.sh --install in this environment. It force- reinstalls torch from the CUDA 12.8 index (torch==2.7.0+cu128) and re-resolves deps, which bumps numpy to 2.x and breaks Isaac Sim 5.1. The plain pip install -e IsaacLab/source/isaaclab above installs Isaac Lab without disturbing the Isaac-Sim-provided torch/numpy stack. If you ever do bump them, restore with: pip install --force-reinstall --index-url https://download.pytorch.org/whl/cu126 torch==2.7.0 torchvision==0.22.0 torchaudio==2.7.0 followed by pip install "numpy==1.26.0" "Pillow==11.3.0" "typing_extensions==4.12.2".

Verify the install (the first Isaac Sim import prompts to accept the Omniverse EULA — set OMNI_KIT_ACCEPT_EULA=YES to accept it non-interactively):

OMNI_KIT_ACCEPT_EULA=YES python -c "import isaacsim, isaaclab; print('Isaac Sim + Isaac Lab OK')"

numpy pin: Isaac Sim 5.1 requires numpy==1.26.0. Installing requirements.txt after Isaac Sim keeps numpy <2; if numpy ever gets bumped to 2.x, Isaac Sim will fail to import — fix with pip install "numpy==1.26.0".

---

3. Usage

Run from the repo root so the extensions package is importable.

Running from another directory or the VSCode debugger? If you hit ModuleNotFoundError: No module named 'extensions', put the repo root on PYTHONPATH. The reproducible way is a conda activation hook so it applies whenever the env is active (terminal and the VSCode debugger):

# ~/miniconda3/envs/point2pose_sdg/etc/conda/activate.d/env_vars.sh
export PYTHONPATH="$HOME/git/Point2Pose-SyntheticDataGenerator:$PYTHONPATH"

Generate synthetic RGB‑D + pose data (2 objects, 3 environments):

python extensions/synthetic_data_generator/run_camera.py --enable_cameras --num_object 2 --num_envs 3

Options:

• --num_object N — number of YCB objects per environment (each randomly chosen).
• --num_envs N — number of parallel environments (each assigned a trajectory type: circle / Y oscillation / Z oscillation / mixed).

Visualize the full YCB set (for figures; press X to save a screenshot):

python extensions/synthetic_data_generator/run_ycb_object_loader.py --enable_cameras

Render the fixed 7-object set from one camera angle to a single PNG (screenshots/ycb_selected_<timestamp>.png), works headless:

python extensions/synthetic_data_generator/run_ycb_selected_shot.py --enable_cameras --headless

Output

run_camera.py writes a timestamped directory under logs/perception/YCB/, one subfolder per environment:

logs/perception/YCB/<timestamp>/env0_<objects>/
├── cam_K.txt                 # 3x3 camera intrinsics
├── rgb/000001.png ...        # RGB (lossless PNG)
├── jpg/000001.jpg ...        # RGB (compressed)
├── depth/000001.png ...      # depth as uint16 (millimeters, depth_scale=1000)
├── normalized_depth/...      # depth normalized to 8-bit for visualization
├── annotated_poses/<object>/000001.txt   # 4x4 object pose in camera frame
└── video.mp4                 # RGB frames assembled into a video

Camera resolution, intrinsics, and pose are configured in extensions/synthetic_data_generator/utils.py (CAMERA_PROPS).

---

License

Portions derived from Isaac Lab are BSD-3-Clause (see file headers).

---

Citation

Point2Pose is published at ECCV 2026. If you use this code, please cite:

@inproceedings{lin2026point2pose,
  title     = {Point2Pose: Occlusion-Recovering 6D Pose Tracking and 3D Reconstruction
               for Multiple Unknown Objects via 2D Point Trackers},
  author    = {Lin, Tzu-Yuan and Lee, Ho Jae and Doherty, Kevin and Lee, Yonghyeon and Kim, Sangbae},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026},
}