Route Experts by Sequence, not by Token

View PDF HTML (experimental)

Abstract:Mixture-of-Experts (MoE) architectures scale large language models (LLMs) by activating only a subset of experts per token, but the standard TopK routing assigns the same fixed number of experts to all tokens, ignoring their varying complexity. Prior adaptive routing methods introduce additional modules and hyperparameters, often requiring costly retraining from scratch. We propose Sequence-level TopK (SeqTopK), a minimal modification that shifts the expert budget from the token level to the sequence level. By selecting the top $T \cdot K$ experts across all $T$ tokens, SeqTopK enables end-to-end learned dynamic allocation -- assigning more experts to difficult tokens and fewer to easy ones -- while preserving the same overall budget. SeqTopK requires only a few lines of code, adds less than 1% overhead, and remains fully compatible with pretrained MoE models. Experiments across math, coding, law, and writing show consistent improvements over TopK and prior parameter-free adaptive methods, with gains that become substantially larger under higher sparsity (up to 16.9%). These results highlight SeqTopK as a simple, efficient, and scalable routing strategy, particularly well-suited for the extreme sparsity regimes of next-generation LLMs. Code is available at this https URL.

Subjects:

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Theory (cs.IT)

Cite as: arXiv:2511.06494 [cs.LG]

(or arXiv:2511.06494v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2511.06494

arXiv-issued DOI via DataCite

Submission history

From: Tiansheng Wen [view email] [v1] Sun, 9 Nov 2025 18:36:07 UTC (1,223 KB) [v2] Fri, 27 Mar 2026 07:05:58 UTC (3,664 KB)