AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation

Authors:Tiancheng Huang, Ruisheng Cao, Yuxin Zhang, Zhangyi Kang, Zijian Wang, Chenrun Wang, Yijie Luo, Hang Zheng, Lirong Qian, Lu Chen, Kai Yu

View PDF HTML (experimental)

Abstract:The growing volume of academic papers has made it increasingly difficult for researchers to efficiently extract key information. While large language models (LLMs) based agents are capable of automating question answering (QA) workflows for scientific papers, there still lacks a comprehensive and realistic benchmark to evaluate their capabilities. Moreover, training an interactive agent for this specific task is hindered by the shortage of high-quality interaction trajectories. In this work, we propose AirQA, a human-annotated comprehensive paper QA dataset in the field of artificial intelligence (AI), with 13,956 papers and 1,246 questions, that encompasses multi-task, multi-modal and instance-level evaluation. Furthermore, we propose ExTrActor, an automated framework for instruction data synthesis. With three LLM-based agents, ExTrActor can perform example generation and trajectory collection without human intervention. Evaluations of multiple open-source and proprietary models show that most models underperform on AirQA, demonstrating the quality of our dataset. Extensive experiments confirm that ExTrActor consistently improves the multi-turn tool-use capability of small models, enabling them to achieve performance comparable to larger ones.

Comments: 29 page, 6 figures, 17 tables, accepted to ICLR 2026

Subjects:

Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

Cite as: arXiv:2509.16952 [cs.CL]

(or arXiv:2509.16952v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2509.16952

arXiv-issued DOI via DataCite

Submission history

From: Tiancheng Huang [view email] [v1] Sun, 21 Sep 2025 07:24:17 UTC (2,855 KB) [v2] Mon, 30 Mar 2026 06:47:00 UTC (3,139 KB)