GenHOI: Generalized Hand-Object Pose Estimation with Occlusion Awareness

View PDF HTML (experimental)

Abstract:Generalized 3D hand-object pose estimation from a single RGB image remains challenging due to the large variations in object appearances and interaction patterns, especially under heavy occlusion. We propose GenHOI, a framework for generalized hand-object pose estimation with occlusion awareness. GenHOI integrates hierarchical semantic knowledge with hand priors to enhance model generalization under challenging occlusion conditions. Specifically, we introduce a hierarchical semantic prompt that encodes object states, hand configurations, and interaction patterns via textual descriptions. This enables the model to learn abstract high-level representations of hand-object interactions for generalization to unseen objects and novel interactions while compensating for missing or ambiguous visual cues. To enable robust occlusion reasoning, we adopt a multi-modal masked modeling strategy over RGB images, predicted point clouds, and textual descriptions. Moreover, we leverage hand priors as stable spatial references to extract implicit interaction constraints. This allows reliable pose inference even under significant variations in object shapes and interaction patterns. Extensive experiments on the challenging DexYCB and HO3Dv2 benchmarks demonstrate that our method achieves state-of-the-art performance in hand-object pose estimation.

Comments: 25 pages, 7 figures

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2603.19013 [cs.CV]

(or arXiv:2603.19013v3 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2603.19013

arXiv-issued DOI via DataCite

Submission history

From: Hui Yang [view email] [v1] Thu, 19 Mar 2026 15:19:23 UTC (1,273 KB) [v2] Fri, 20 Mar 2026 10:37:04 UTC (1,273 KB) [v3] Sun, 29 Mar 2026 05:50:06 UTC (1,273 KB)