FinTruthQA: A Benchmark for AI-Driven Financial Disclosure Quality Assessment in Investor -- Firm Interactions

View PDF HTML (experimental)

Abstract:Accurate and transparent financial information disclosure is essential for market efficiency, investor decision-making, and corporate governance. Chinese stock exchanges' investor interactive platforms provide a widely used channel through which listed firms respond to investor concerns, yet these responses are often limited or non-substantive, making disclosure quality difficult to assess at scale. To address this challenge, we introduce FinTruthQA, to our knowledge the first benchmark for AI-driven assessment of financial disclosure quality in investor-firm interactions. FinTruthQA comprises 6,000 real-world financial Q&A entries, each manually annotated based on four key evaluation criteria: question identification, question relevance, answer readability, and answer relevance. We benchmark statistical machine learning models, pre-trained language models and their fine-tuned variants, as well as large language models (LLMs), on FinTruthQA. Experiments show that existing models achieve strong performance on question identification and question relevance (F1 > 95%), but remain substantially weaker on answer readability (Micro F1 approximately 88%) and especially answer relevance (Micro F1 approximately 80%), highlighting the nontrivial difficulty of fine-grained disclosure quality assessment. Domain- and task-adapted pre-trained language models consistently outperform general-purpose models and LLM-based prompting on the most challenging settings. These findings position FinTruthQA as a practical foundation for AI-driven disclosure monitoring in capital markets, with value for regulatory oversight, investor protection, and disclosure governance in real-world financial settings.

Subjects:

Computation and Language (cs.CL)

Cite as: arXiv:2406.12009 [cs.CL]

(or arXiv:2406.12009v4 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2406.12009

arXiv-issued DOI via DataCite

Submission history

From: Ziyue Xu [view email] [v1] Mon, 17 Jun 2024 18:25:02 UTC (1,216 KB) [v2] Sat, 7 Dec 2024 15:47:26 UTC (1,273 KB) [v3] Tue, 11 Feb 2025 16:49:17 UTC (1,042 KB) [v4] Fri, 27 Mar 2026 08:49:53 UTC (778 KB)