EasyHeC: Accurate and Automatic Hand-eye Calibration via
Differentiable Rendering and Space Exploration

Linghao Chen1,2, Yuzhe Qin2, Xiaowei Zhou1, Hao Su2

1State Key Lab of CAD & CG, Zhejiang University
2University of California, San Diego


Hand-eye calibration is a critical task in robotics, as it directly affects the efficacy of critical operations such as manipulation and grasping. Traditional methods for achieving this objective necessitate the careful design of joint poses and the use of specialized calibration markers, while most recent learning-based approaches using solely pose regression are limited in their abilities to diagnose inaccuracies. In this work, we introduce a new approach to hand-eye calibration called EasyHeC, which is markerless, white-box, and offers comprehensive coverage of positioning accuracy across the entire robot configuration space. We introduce two key technologies: differentiable rendering-based camera pose optimization and consistency-based joint space exploration, which enables accurate end-to-end optimization of the calibration process and eliminates the need for the laborious manual design of robot joint poses. Our evaluation demonstrates superior performance in synthetic and real-world datasets, enhancing downstream manipulation tasks by providing precise camera poses for locating and interacting with objects.

Overview video

Accurate calibration via differentiable rendering

Automatic calibration via joint space exploration

Real-world applications powered by EasyHeC


  title={EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration},
  author={Chen, Linghao and Qin, Yuzhe and Zhou, Xiaowei and Su, Hao},
  journal={arXiv preprint},