EasyHeC++: Fully Automatic Hand-Eye Calibration
with Pretrained Image Models

Linghao Chen1,2*, Kangfu Zheng3*, Zhengdong Hong1,2*, Xiaowei Zhou1, Hao Su2

1Zhejiang University 2UC San Diego 3Tsinghua University
*Equal contribution


Hand-eye calibration plays a fundamental role in robotics by directly influencing the efficiency of critical operations such as manipulation and grasping. In this work, we present a novel framework, EasyHeC++, designed for fully automatic hand-eye calibration. In contrast to previous methods that necessitate manual calibration, specialized markers, or the training of arm-specific neural networks, our approach is the first system that enables accurate calibration of any robot arm in a marker-free, training-free, and fully automatic manner. Our approach employs a two-step process. First, we initialize the camera pose using a sampling or feature-matching-based method with the aid of pretrained image models. Subsequently, we perform pose optimization through differentiable rendering. Extensive experiments demonstrate the system’s superior accuracy in both synthetic and real-world datasets across various robot arms and camera settings. The code will be publicly available upon the publication of this paper.

Comparison against previous methods

Overview video


  title={Fully Automatic Hand-Eye Calibration with Pretrained Image Models},
  author={Chen, Linghao and Zheng, Kangfu and Hong, Zhengdong and Zhou, Xiaowei and Su, Hao},
  journal={arXiv preprint},