QPT V2: Masked Image Modeling Advances Visual Scoring

View PDF HTML (experimental)

Abstract:Quality assessment and aesthetics assessment aim to evaluate the perceived quality and aesthetics of visual content. Current learning-based methods suffer greatly from the scarcity of labeled data and usually perform sub-optimally in terms of generalization. Although masked image modeling (MIM) has achieved noteworthy advancements across various high-level tasks (e.g., classification, detection etc.). In this work, we take on a novel perspective to investigate its capabilities in terms of quality- and aesthetics-awareness. To this end, we propose Quality- and aesthetics-aware pretraining (QPT V2), the first pretraining framework based on MIM that offers a unified solution to quality and aesthetics assessment. To perceive the high-level semantics and fine-grained details, pretraining data is curated. To comprehensively encompass quality- and aesthetics-related factors, degradation is introduced. To capture multi-scale quality and aesthetic information, model structure is modified. Extensive experimental results on 11 downstream benchmarks clearly show the superior performance of QPT V2 in comparison with current state-of-the-art approaches and other pretraining paradigms.

Comments: 8 pages, 6 figures. Accepted by ACM MM 24

Subjects:

Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM)

Cite as: arXiv:2407.16541 [cs.CV]

(or arXiv:2407.16541v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2407.16541

arXiv-issued DOI via DataCite

Submission history

From: Qizhi Xie [view email] [v1] Tue, 23 Jul 2024 14:53:47 UTC (6,312 KB) [v2] Fri, 27 Mar 2026 03:13:37 UTC (4,581 KB)