A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Models

View PDF

Abstract:Despite the rapid progress in multimodal models and Large Visual-Language Models (LVLM), they remain highly susceptible to adversarial perturbations, raising serious concerns about their reliability in real-world use. While adversarial training has become the leading paradigm for building models that are robust to adversarial attacks, Test-Time Transformations (TTT) have emerged as a promising strategy to boost robustness at this http URL light of this, we propose Energy-Guided Test-Time Transformation (ET3), a lightweight, training-free defense that enhances the robustness by minimizing the energy of the input this http URL method is grounded in a theory that proves our transformation succeeds in classification under reasonable assumptions. We present extensive experiments demonstrating that ET3 provides a strong defense for classifiers, zero-shot classification with CLIP, and also for boosting the robustness of LVLMs in tasks such as Image Captioning and Visual Question Answering. Code is available at this http URL .

Comments: Accepted at the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2026, Main Conference

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2603.26984 [cs.CV]

(or arXiv:2603.26984v1 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2603.26984

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Mujtaba Hussain Mirza [view email] [v1] Fri, 27 Mar 2026 20:53:04 UTC (18,805 KB)