Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions

View PDF HTML (experimental)

Abstract:Model merging combines the parameters of multiple neural networks into a single model without additional training. As fine-tuned large language models (LLMs) proliferate, merging offers a computationally efficient alternative to ensembles and full retraining, enabling practitioners to compose specialized capabilities at minimal cost. This survey examines model merging in the LLM era through the \textbf{FUSE} taxonomy, organized along \textbf{F}oundations, \textbf{U}nification Strategies, \textbf{S}cenarios, and \textbf{E}cosystem. We first establish the theoretical underpinnings of merging, including loss landscape geometry and mode connectivity, then systematically review the algorithmic space spanning weight averaging, task vector arithmetic, sparsification-enhanced methods, mixture-of-experts architectures, and evolutionary optimization. We further examine downstream applications across multi-task learning, safety alignment, domain specialization, and federated learning, and survey the supporting ecosystem of tools and evaluation benchmarks. Finally, we identify key open challenges and future directions, aiming to equip researchers and practitioners with a structured foundation for advancing model merging.

Subjects:

Computation and Language (cs.CL)

Cite as: arXiv:2603.09938 [cs.CL]

(or arXiv:2603.09938v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2603.09938

arXiv-issued DOI via DataCite

Submission history

From: Mingyang Song [view email] [v1] Tue, 10 Mar 2026 17:31:55 UTC (105 KB) [v2] Mon, 30 Mar 2026 16:23:51 UTC (132 KB)