Data Organization Matters in Multimodal Instruction Tuning: A Controlled Study of Capability Trade-offs

View PDF HTML (experimental)

Abstract:Recent multimodal large language models (MLLMs) perform strongly on general visual understanding, diagram and chart reasoning, and document-centric perception. However, these abilities are learned from heterogeneous supervision sources with very different task structures and learning demands, and the effect of their temporal organization during training remains underexplored. We study whether data organization affects the trade-off among general understanding, structured reasoning, and fine-grained OCR/document understanding in multimodal instruction tuning. To isolate this factor, we use a controlled three-stage training framework in which the backbone, trainable modules, and optimization pipeline are fixed across all runs, and only the temporal arrangement of post-alignment supervision is changed. We compare four strategies: direct mixture, curriculum training, balanced sampling, and reverse curriculum. Experiments on general visual instruction following, diagram reasoning, chart reasoning, scene-text question answering, and document question answering show that data organization is a first-order design variable in multimodal adaptation. Curriculum training gives the best overall trade-off and the strongest structured reasoning performance. Balanced sampling is better for OCR-oriented capability but weakens the broader capability balance. Reverse curriculum performs worst in both final performance and optimization stability. Training-dynamics analysis further suggests that building general understanding and reasoning before introducing OCR-intensive supervision leads to smoother optimization and faster convergence. These findings highlight data scheduling as an explicit design dimension for multimodal model adaptation.

Comments: 12 pages, 2 figures

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

ACM classes: I.2.6; I.2.10

Cite as: arXiv:2603.27744 [cs.CV]

(or arXiv:2603.27744v1 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2603.27744

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Guowei Tang [view email] [v1] Sun, 29 Mar 2026 15:54:06 UTC (276 KB)