Attention-Aligned Reasoning for Large Language Models

View PDF HTML (experimental)

Abstract:Large Language Models (LLMs) tend to generate a long reasoning chain when solving complex tasks. However, as the reasoning chain extends, critical intermediate steps and the original prompt will be buried in the context, receiving insufficient attention and leading to errors. In this work, we present ATAR, a novel reasoning method that leverages the inherent reasoning structure to steer LLM attention. Our experiments show that ATAR outperforms SOTA methods across six benchmarks, achieving up to 15.39% absolute improvement. Furthermore, with ATAR, "non-reasoning" models achieve comparable or even better performance compared to reasoning models of the same size in most benchmarks. Finally, our ablation studies show that the attention alignment component contributes significantly, and that these improvements are persist under different attentionsteering backends.

Subjects:

Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

Cite as: arXiv:2510.03223 [cs.CL]

(or arXiv:2510.03223v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2510.03223

arXiv-issued DOI via DataCite

Submission history

From: Hongxiang Zhang [view email] [v1] Fri, 3 Oct 2025 17:56:33 UTC (791 KB) [v2] Fri, 27 Mar 2026 17:14:20 UTC (1,086 KB)