Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs

View PDF HTML (experimental)

Abstract:Arithmetic benchmarks are often reduced to a single held-out score, but that score can conflate qualitatively different failures. We study a controlled minimal GPT trained on exhaustive 2-digit addition, where all local digit transitions are already present in training, and ask why 3-digit generalization still fails. The failure is staged. First, there is a layout barrier: a learned absolute-position model collapses under a pure 3-digit layout shift, and mixed-layout exposure is the only intervention that materially weakens this barrier. Second, after layout repair, the hundreds position behaves like a carry flag rather than a semantic hundreds digit; targeted carry probes reverse the relevant logit margin, whereas a matched extra-data control does not. Third, after carry repair, the main remaining bottleneck is conditional recomposition: high-conditioned tail data outperforms a matched control, high-only data, and tail-only data on all true-3-digit suites, and the same ordering reappears in a larger 2-layer bridge experiment. The residual errors after recomposition are then overwhelmingly tens-only, and a separate 10-seed late-stage study shows that a sign-aware tens repair raises exact match on the hardest thousands-carry suite from 0.664 to 0.822. We therefore provide an experimentally testable decomposition of arithmetic OOD failure into layout, carry-semantics, recomposition, and late tens-residual stages.

Comments: 16 pages, 4 figures

Subjects:

Computation and Language (cs.CL); Machine Learning (cs.LG)

ACM classes: I.2.7

Cite as: arXiv:2603.26828 [cs.CL]

(or arXiv:2603.26828v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2603.26828

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Seine A. Shintani [view email] [v1] Fri, 27 Mar 2026 03:40:27 UTC (1,038 KB)