Privacy-Accuracy Trade-offs in High-Dimensional LASSO under Perturbation Mechanisms

View PDF HTML (experimental)

Abstract:We study privacy-preserving sparse linear regression in the high-dimensional regime, focusing on the LASSO estimator. We analyze two widely used mechanisms for differential privacy: output perturbation, which injects noise into the estimator, and objective perturbation, which adds a random linear term to the loss function. Using approximate message passing (AMP), we characterize the typical behavior of these estimators under random design and privacy noise. To quantify privacy, we adopt typical-case measures, including the on-average KL divergence, which admits a hypothesis-testing interpretation in terms of distinguishability between neighboring datasets. Our analysis reveals that sparsity plays a central role in shaping the privacy-accuracy trade-off: stronger regularization can improve privacy by stabilizing the estimator against single-point data changes. We further show that the two mechanisms exhibit qualitatively different behaviors. In particular, for objective perturbation, increasing the noise level can have non-monotonic effects, and excessive noise may destabilize the estimator, leading to increased sensitivity to data perturbations. Our results demonstrate that AMP provides a powerful framework for analyzing privacy-accuracy trade-offs in high-dimensional sparse models.

Comments: 53 pages, 11 figures

Subjects:

Machine Learning (stat.ML); Machine Learning (cs.LG)

Cite as: arXiv:2603.26227 [stat.ML]

(or arXiv:2603.26227v1 [stat.ML] for this version)

https://doi.org/10.48550/arXiv.2603.26227

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Ayaka Sakata [view email] [v1] Fri, 27 Mar 2026 09:50:43 UTC (254 KB)