When Surfaces Lie: Exploiting Wrinkle-Induced Attention Shift to Attack Vision-Language Models

View PDF HTML (experimental)

Abstract:Visual-Language Models (VLMs) have demonstrated exceptional cross-modal understanding across various tasks, including zero-shot classification, image captioning, and visual question answering. However, their robustness to physically plausible non-rigid deformations-such as wrinkles on flexible surfaces-remains poorly understood. In this work, we propose a parametric structural perturbation method inspired by the mechanics of three-dimensional fabric wrinkles. Specifically, our method generates photorealistic non-rigid perturbations by constructing multi-scale wrinkle fields and integrating displacement field distortion with surface-consistent appearance variations. To achieve an optimal balance between visual naturalness and adversarial effectiveness, we design a hierarchical fitness function in a low-dimensional parameter space and employ an optimization-based search strategy. We evaluate our approach using a two-stage framework: perturbations are first optimized on a zero-shot classification proxy task and subsequently assessed for transferability on generative tasks. Experimental results demonstrate that our method significantly degrades the performance of various state-of-the-art VLMs, consistently outperforming baselines in both image captioning and visual question-answering tasks.

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2603.27759 [cs.CV]

(or arXiv:2603.27759v1 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2603.27759

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Xiang Chen [view email] [v1] Sun, 29 Mar 2026 16:35:18 UTC (6,762 KB)