Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models

View PDF HTML (experimental)

Abstract:Multimodal Large Language Models (MLLMs) perform strong vision-language reasoning under standard conditions but fail in extreme illumination, where RGB inputs lose irrevocable structure and semantics. We propose Event-MLLM, an event-enhanced model that performs all-light visual reasoning by dynamically fusing event streams with RGB frames. Two key components drive our approach: an Illumination Indicator - a learnable signal derived from a DINOv2 branch that represents exposure degradation and adaptively modulates event-RGB fusion - and an Illumination Correction Loss that aligns fused features with non-degraded (normal-light) semantics in the latent space, compensating for information lost in extreme lighting. We curate the first multi-illumination event-instruction corpus for MLLMs, with 2,241 event-RGB samples (around 6 QA pairs each) across diverse scenes and 17 brightness rates (0.05x - 20x), plus an instruct-following benchmark for reasoning, counting, and fine-grained recognition under extreme lighting. Experiments show that Event-MLLM markedly outperforms general-purpose, illumination-adaptive, and event-only baselines, setting a new state of the art in robust multimodal perception and reasoning under challenging illumination.

Comments: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2026

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2603.27558 [cs.CV]

(or arXiv:2603.27558v1 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2603.27558

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Jiahui Liu [view email] [v1] Sun, 29 Mar 2026 07:46:32 UTC (1,879 KB)