TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba

View PDF HTML (experimental)

Abstract:Music-to-dance generation has broad applications in virtual reality, dance education, and digital character animation. However, the limited coverage of existing 3D dance datasets confines current models to a narrow subset of music styles and choreographic patterns, resulting in poor generalization to real-world music. Consequently, generated dances often become overly simplistic and repetitive, substantially degrading expressiveness and realism. To tackle this problem, we present TokenDance, a two-stage music-to-dance generation framework that explicitly addresses this limitation through dual-modality tokenization and efficient token-level generation. In the first stage, we discretize both dance and music using Finite Scalar Quantization, where dance motions are factorized into upper and lower-body components with kinematic-dynamic constraints, and music is decomposed into semantic and acoustic features with dedicated codebooks to capture choreography-specific structures. In the second stage, we introduce a Local-Global-Local token-to-token generator built on a Bidirectional Mamba backbone, enabling coherent motion synthesis, strong music-dance alignment, and efficient non-autoregressive inference. Extensive experiments demonstrate that TokenDance achieves overall state-of-the-art (SOTA) performance in both generation quality and inference speed, highlighting its effectiveness and practical value for real-world music-to-dance applications.

Comments: CVPR2026 Workshop on HuMoGen

Subjects:

Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Sound (cs.SD)

Cite as: arXiv:2603.27314 [cs.AI]

(or arXiv:2603.27314v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2603.27314

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Ziyue Yang [view email] [v1] Sat, 28 Mar 2026 15:38:14 UTC (13,856 KB)