LaSM: Layer-wise Scaling Mechanism for Defending Pop-up Attack on GUI Agents

View PDF HTML (experimental)

Abstract:Graphical user interface (GUI) agents built on multimodal large language models (MLLMs) have recently demonstrated strong decision-making abilities in screen-based interaction tasks. However, they remain highly vulnerable to pop-up-based environmental injection attacks, where malicious visual elements divert model attention and lead to unsafe or incorrect actions. Existing defense methods either require costly retraining or perform poorly under inductive interference. In this work, we systematically study how such attacks alter the attention behavior of GUI agents and uncover a layer-wise attention divergence pattern between correct and incorrect outputs. Based on this insight, we propose \textbf{LaSM}, a \textit{Layer-wise Scaling Mechanism} that selectively amplifies attention and MLP modules in critical layers. LaSM improves the alignment between model saliency and task-relevant regions without additional training. Extensive experiments across multiple datasets demonstrate that our method significantly improves the defense success rate and exhibits strong robustness, while having negligible impact on the model's general capabilities. Our findings reveal that attention misalignment is a core vulnerability in MLLM agents and can be effectively addressed through selective layer-wise modulation. Our code can be found in this https URL.

Subjects:

Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)

Cite as: arXiv:2507.10610 [cs.CR]

(or arXiv:2507.10610v2 [cs.CR] for this version)

https://doi.org/10.48550/arXiv.2507.10610

arXiv-issued DOI via DataCite

Submission history

From: Zihe Yan [view email] [v1] Sun, 13 Jul 2025 08:36:09 UTC (3,121 KB) [v2] Tue, 31 Mar 2026 08:10:46 UTC (19,066 KB)