Gesture-Aware Pretraining and Token Fusion for 3D Hand Pose Estimation

View PDF HTML (experimental)

Abstract:Estimating 3D hand pose from monocular RGB images is fundamental for applications in AR/VR, human-computer interaction, and sign language understanding. In this work we focus on a scenario where a discrete set of gesture labels is available and show that gesture semantics can serve as a powerful inductive bias for 3D pose estimation. We present a two-stage framework: gesture-aware pretraining that learns an informative embedding space using coarse and fine gesture labels from InterHand2.6M, followed by a per-joint token Transformer guided by gesture embeddings as intermediate representations for final regression of MANO hand parameters. Training is driven by a layered objective over parameters, joints, and structural constraints. Experiments on InterHand2.6M demonstrate that gesture-aware pretraining consistently improves single-hand accuracy over the state-of-the-art EANet baseline, and that the benefit transfers across architectures without any modification.

Comments: 6 pages, 6 figures

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2603.17396 [cs.CV]

(or arXiv:2603.17396v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2603.17396

arXiv-issued DOI via DataCite

Submission history

From: Rui Hong [view email] [v1] Wed, 18 Mar 2026 06:18:46 UTC (5,961 KB) [v2] Sun, 29 Mar 2026 01:48:18 UTC (5,961 KB)