From Exploration to Exploitation: A Two-Stage Entropy RLVR Approach for Noise-Tolerant MLLM Training
arXiv:2511.07738v2 Announce Type: replace Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) for Multimodal Large Language Models (MLLMs) is highly dependent on high-quality labeled data, which is often scarce and prone to substantial annotation noise in real-world scenarios. Existing unsupervised RLVR methods, including pure entropy minimization, can overfit to incorrect labels and limit the crucial reward ranking signal for Group-Relative Policy Optimization (GRPO). To address these challenges and enhance noise tolerance, we propose a novel two-stage, token-level entropy optimiz — Donglai Xu, Hongzheng Yang, Yuzhi Zhao, Pingping Zhang, Jinpeng Chen, Wenao Ma, Zhijian Hou, Mengyang Wu, Xiaolei Li, Senkang Hu, Ziyi Guan, Jason Chun Lok Li, Lai Man Po
Authors:Donglai Xu, Hongzheng Yang, Yuzhi Zhao, Pingping Zhang, Jinpeng Chen, Wenao Ma, Zhijian Hou, Mengyang Wu, Xiaolei Li, Senkang Hu, Ziyi Guan, Jason Chun Lok Li, Lai Man Po
View PDF HTML (experimental)
Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) for Multimodal Large Language Models (MLLMs) is highly dependent on high-quality labeled data, which is often scarce and prone to substantial annotation noise in real-world scenarios. Existing unsupervised RLVR methods, including pure entropy minimization, can overfit to incorrect labels and limit the crucial reward ranking signal for Group-Relative Policy Optimization (GRPO). To address these challenges and enhance noise tolerance, we propose a novel two-stage, token-level entropy optimization method for RLVR. This approach dynamically guides the model from exploration to exploitation during training. In the initial exploration phase, token-level entropy maximization promotes diverse and stochastic output generation, serving as a strong regularizer that prevents premature convergence to noisy labels and ensures sufficient intra-group variation, which enables more reliable reward gradient estimation in GRPO. As training progresses, the method transitions into the exploitation phase, where token-level entropy minimization encourages the model to produce confident and deterministic outputs, thereby consolidating acquired knowledge and refining prediction accuracy. Empirically, across three MLLM backbones - Qwen2-VL-2B, Qwen2-VL-7B, and Qwen2.5-VL-3B - spanning diverse noise settings and multiple tasks, our phased strategy consistently outperforms prior approaches by unifying and enhancing external, internal, and entropy-based methods, delivering robust and superior performance across the board.
Subjects:
Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
Cite as: arXiv:2511.07738 [cs.LG]
(or arXiv:2511.07738v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2511.07738
arXiv-issued DOI via DataCite
Submission history
From: Donglai Xu [view email] [v1] Tue, 11 Nov 2025 01:42:37 UTC (421 KB) [v2] Sat, 28 Mar 2026 07:59:38 UTC (1,963 KB)
Sign in to highlight and annotate this article
Conversation starters
Daily AI Digest
Get the top 5 AI stories delivered to your inbox every morning.
More about
researchpaperarxiv
Simulating Realistic LiDAR Data Under Adverse Weather for Autonomous Vehicles: A Physics-Informed Learning Approach
arXiv:2604.01254v1 Announce Type: cross Abstract: Accurate LiDAR simulation is crucial for autonomous driving, especially under adverse weather conditions. Existing methods struggle to capture the complex interactions between LiDAR signals and atmospheric phenomena, leading to unrealistic representations. This paper presents a physics-informed learning framework (PICWGAN) for generating realistic LiDAR data under adverse weather conditions. By integrating physicsdriven constraints for modeling signal attenuation and geometryconsistent degradations into a physics-informed learning pipeline, the proposed method reduces the sim-to-real gap. Evaluations on real-world datasets (CADC for snow, Boreas for rain) and the VoxelScape dataset show that our approach closely mimics realworld intensity p
Knowledge Map
Connected Articles — Knowledge Graph
This article is connected to other articles through shared AI topics and tags.
More in Research Papers
Simulating Realistic LiDAR Data Under Adverse Weather for Autonomous Vehicles: A Physics-Informed Learning Approach
arXiv:2604.01254v1 Announce Type: cross Abstract: Accurate LiDAR simulation is crucial for autonomous driving, especially under adverse weather conditions. Existing methods struggle to capture the complex interactions between LiDAR signals and atmospheric phenomena, leading to unrealistic representations. This paper presents a physics-informed learning framework (PICWGAN) for generating realistic LiDAR data under adverse weather conditions. By integrating physicsdriven constraints for modeling signal attenuation and geometryconsistent degradations into a physics-informed learning pipeline, the proposed method reduces the sim-to-real gap. Evaluations on real-world datasets (CADC for snow, Boreas for rain) and the VoxelScape dataset show that our approach closely mimics realworld intensity p
MIMO Capacity Enhancement by Grating Walls: A Physics-Based Proof of Principle
arXiv:2604.01786v1 Announce Type: new Abstract: This paper investigates the passive enhancement of MIMO spectral efficiency through boundary engineering in a simplified two dimensional indoor proof of principle model. The propagation channel is constructed from the electromagnetic Green's function of a room with boundaries modeled as free space, drywall, perfect electric conductor (PEC), or binary gratings. Within this framework, grating coated walls enrich the non line of sight (NLoS) multipath field, reduce channel correlation, and enhance spatial multiplexing over a broad range of receiver locations. Comparisons with the drywall and PEC reference cases further reveal that the observed capacity enhancement arises not from diffraction alone, but from the combined effects of effective wall
1-bit Quantized Continuous Aperture Arrays
arXiv:2604.01780v1 Announce Type: new Abstract: Continuous aperture arrays (CAPAs) have emerged as a promising physical-layer paradigm for sixth generation (6G) systems, offering spatial degrees of freedom beyond those of conventional discrete antenna arrays. This paper investigates the interaction between the CAPA receive architecture and low-cost 1-bit analog-to-digital converters (ADCs), which impose a severe nonlinear distortion penalty in conventional discrete systems. For Rayleigh fading, we derive a moment matching approximation (MMA)-based closed-form symbol error probability (SEP) approximation based on Gamma moment-matching of the spatial eigenvalue distribution, and show that CAPAs incur a diversity-order penalty governed by Jensen's inequality on the mode eigenvalues. For line-
Discussion
Sign in to join the discussion
No comments yet — be the first to share your thoughts!