On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models

Multimodal Continual Instruction Tuning aims to continually enhance Large Vision-Language Models by learning from new data without forgetting previously acquired knowledge. Dynamic MoE architectures naturally facilitate this by incrementally adding new experts while keeping existing ones frozen. However, despite expert isolation, MoE-based continual learners still suffer from forgetting due to routing drift: old-task tokens become mistakenly attracted to newly added experts.

This paper analyzes the failure mode at the token level and reveals the Token's Dilemma: ambiguous and old tokens in new-task data offer minimal learning benefit yet induce forgetting when routed to new experts. Motivated by this, the authors propose LLaVA-DyMoE, a dynamic MoE framework with drift-aware token assignment that steers problematic tokens away from new experts and enforces expert-group separation through targeted regularization — with no inference overhead and orthogonal to existing CL strategies.