THEval. Evaluation Framework for Talking Head Video Generation

View PDF HTML (experimental)

Abstract:Video generation has achieved remarkable progress, with generated videos increasingly resembling real ones. However, the rapid advance in generation has outpaced the development of adequate evaluation metrics. Currently, the assessment of talking head generation primarily relies on limited metrics, evaluating general video quality, lip synchronization, and on conducting user studies. Motivated by this, we propose a new evaluation framework comprising 8 metrics related to three dimensions (i) quality, (ii) naturalness, and (iii) synchronization. In selecting the metrics, we place emphasis on efficiency, as well as alignment with human preferences. Based on this considerations, we streamline to analyze fine-grained dynamics of head, mouth, and eyebrows, as well as face quality. Our extensive experiments on 85,000 videos generated by 17 state-of-the-art models suggest that while many algorithms excel in lip synchronization, they face challenges with generating expressiveness and artifact-free details. These videos were generated based on a novel real dataset, that we have curated, in order to mitigate bias of training data. Our proposed benchmark framework is aimed at evaluating the improvement of generative methods. Original code, dataset and leaderboards will be publicly released and regularly updated with new methods, in order to reflect progress in the field.

Comments: CVPR 2025 Findings, Project Page: this https URL

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2511.04520 [cs.CV]

(or arXiv:2511.04520v3 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2511.04520

arXiv-issued DOI via DataCite

Submission history

From: Nabyl Quignon [view email] [v1] Thu, 6 Nov 2025 16:34:10 UTC (3,425 KB) [v2] Fri, 7 Nov 2025 14:25:58 UTC (3,425 KB) [v3] Sun, 29 Mar 2026 17:57:46 UTC (3,404 KB)