MuCaPoser is a full-body self-avatar tracking system for VR that combines multiview RGB pose estimation with sparse HMD tracking. Using a multiview camera rig alongside the headset's own 6-DoF pose and controller/hand tracking, the system estimates and animates a full-body avatar in real time without requiring external body-worn trackers.
| Submodule | Type | Role |
|---|---|---|
HMDCameraCalibrationUnity |
Unity | Captures a checkerboard snapshot through the HMD's passthrough cameras alongside HMD pose |
HMDCameraCalibrationCore |
Python | Processes the checkerboard snapshot to compute the extrinsics of the RGB cameras relative to the HMD |
DeepPoserTraining |
Python | Deep-learning training procedure for the self-avatar animation model |
HMDCameraAnimationUnity |
Unity | Runtime self-avatar animation: consumes RGB camera streams + HMD/controller tracking, runs the trained model, and drives the avatar |
Each submodule is an independent git repository with its own history, included here as a git submodule.
Clone the repository and its submodules:
git clone --recurse-submodules https://github.com/antmaio/MuCHMDPoser.gitFollow the instructions in the submodules for each respective task.
HMDCameraCalibrationUnitycaptures a checkerboard snapshot from the HMD's passthrough RGB cameras.HMDCameraCalibrationCoreprocesses that snapshot to compute the extrinsic calibration between each RGB camera and the HMD tracking frame.DeepPoserTrainingtrains the pose-fusion model offline, combining multiview RGB pose estimation with sparse HMD/controller tracking signals, and exports trained weights.HMDCameraAnimationUnityloads the calibration output and trained weights at runtime to animate the user's full-body self-avatar live, inside the headset.
@article{maiorca2026muchmdposer,
title={MuCHMDPoser: Multiview RGB-Camera Full-Body Pose Tracking for Virtual Reality},
author={Maiorca, Antoine and Fletcher, George and Ravet, Thierry and Dutoit, Thierry},
journal={arXiv preprint arXiv:2611.03062},
year={2026}
}