Detecting Call Graph Unsoundness without Ground Truth

View PDF HTML (experimental)

Abstract:Java static analysis frameworks are commonly compared under the assumption that analysis algorithms and configurations compose monotonically and yield semantically comparable results across tools. In this work, we show that this assumption is fundamentally flawed. We present a large-scale empirical study of semantic consistency within and across four widely used Java static analysis frameworks: Soot, SootUp, WALA, and Doop. Using precision partial orders over analysis algorithms and configurations, we systematically identify violations where increased precision introduces new call-graph edges or amplifies inconsistencies. Our results reveal three key findings. First, algorithmic precision orders frequently break within frameworks due to modern language features such as lambdas, reflection, and native modeling. Second, configuration choices strongly interact with analysis algorithms, producing synergistic failures that exceed the effects of algorithm or configuration changes alone. Third, cross-framework comparisons expose irreconcilable semantic gaps, demonstrating that different frameworks operate over incompatible notions of call-graph ground truth. These findings challenge prevailing evaluation practices in static analysis and highlight the need to reason jointly about algorithms, configurations, and framework semantics when assessing precision and soundness.

Subjects:

Software Engineering (cs.SE)

Cite as: arXiv:2604.00885 [cs.SE]

(or arXiv:2604.00885v1 [cs.SE] for this version)

https://doi.org/10.48550/arXiv.2604.00885

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Fangtian Zhong [view email] [v1] Wed, 1 Apr 2026 13:32:53 UTC (209 KB)