UVR Lab. was formed in Feb. 2001 at GIST to study and develop “Virtual Reality in Smart computing environments” that process multimodal input, perceive user’s intention and emotion, and respond to user’s request through Augmented Reality. Since 2012, UVR Lab moved to KAIST GSCT and restarted with a theme of “FUN in Ubiquitous VR.”
작성일 : 21-05-25
3D Hand Pose Estimation with a Single Infrared Camera via Domain Transfer Learning
 글쓴이 : UVR
조회 : 1,779  
Gabyong Park, Tae-Kyun Kim, Woontack Woo. “3D Hand Pose Estimation with a Single Infrared Camera via Domain Transfer Learning.", 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2020.

Previous methods successfully estimated 3D hand poses from unblurred depth images with slow and smooth hand motions. However, the performance drops when the depth images are contaminated by motion blur due to fast hand motion. In this paper, we exploit an infrared (IR) image input, which is weakly blurred under fast hand motion. The proposed method is based on domain transfer learning from depth to infrared images. Note we do not have IR images with hand skeletons, thus proposing self-supervision rather than direct supervision using the skeleton labels. We train a Hand Image Generator (HIG) and two Hand Pose Estimators (HPEs) on paired depth and infrared images via self-supervision using a consistency loss, guided by an existing HPE trained on paired depth and hand skeleton entries. The IR-based HPE is then refined on the weakly blurred infrared images. The qualitative and quantitative experiments demonstrate that the proposed method accurately estimates 3D hand poses under motion blur by fast hand motion, while existing depth-based methods fail. Our solution therefore supports fast 3D manipulation of virtual objects for augmented reality applications. Our model and dataset are publicly available for future research.


ADD. (34141)KAIST N5 2325, 291 Daehak-ro, Yuseong-gu, Daejeon, Republic of Korea / TEL. +82-42-350-5923