Event-based Facial Keypoint Alignment via Cross-Modal Fusion Attention and Self-Supervised Multi-Event Representation Learning

View PDF HTML (experimental)

Abstract:Event cameras offer unique advantages for facial keypoint alignment under challenging conditions, such as low light and rapid motion, due to their high temporal resolution and robustness to varying illumination. However, existing RGB facial keypoint alignment methods do not perform well on event data, and training solely on event data often leads to suboptimal performance because of its limited spatial information. Moreover, the lack of comprehensive labeled event datasets further hinders progress in this area. To address these issues, we propose a novel framework based on cross-modal fusion attention (CMFA) and self-supervised multi-event representation learning (SSMER) for event-based facial keypoint alignment. Our framework employs CMFA to integrate corresponding RGB data, guiding the model to extract robust facial features from event input images. In parallel, SSMER enables effective feature learning from unlabeled event data, overcoming spatial limitations. Extensive experiments on our real-event E-SIE dataset and a synthetic-event version of the public WFLW-V benchmark show that our approach consistently surpasses state-of-the-art methods across multiple evaluation metrics.

Comments: 14 pages, 10 figures

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2509.24968 [cs.CV]

(or arXiv:2509.24968v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2509.24968

arXiv-issued DOI via DataCite

Submission history

From: DongHwa Kang [view email] [v1] Mon, 29 Sep 2025 16:00:50 UTC (7,382 KB) [v2] Fri, 27 Mar 2026 18:02:39 UTC (9,097 KB)