Linear Discriminant Analysis with Gradient Optimization

View PDF HTML (experimental)

Abstract:Linear discriminant analysis (LDA) is a fundamental classification and dimension reduction method that achieves Bayes optimality under Gaussian mixture, but often struggles in high-dimensional settings where the covariance matrix cannot be reliably estimated. We propose LDA with gradient optimization (LDA-GO), which learns a low-rank precision matrix via scalable gradient-based optimization. The method automatically selects between a Gaussian likelihood and a cross-entropy loss using data-driven structural diagnostics, adapting to the signal structure without user tuning. The gradient computation avoids any quadratic-sized intermediate matrix, keeping the per-iteration cost linear in the number of dimensions. Theoretically, we prove several properties of the method, including the convexity of the objective functions, Bayes-optimality of the method, and a finite-sample bound of the excess error. Numerically, we conducted a variety of simulations and real data experiments to show that LDA-GO wins a majority of settings among other LDA variants, particularly in sparse-signal high-dimensional regimes.

Comments: 26 pages

Subjects:

Computation (stat.CO); Machine Learning (stat.ML)

Cite as: arXiv:2506.06845 [stat.CO]

(or arXiv:2506.06845v2 [stat.CO] for this version)

https://doi.org/10.48550/arXiv.2506.06845

arXiv-issued DOI via DataCite

Submission history

From: Cencheng Shen [view email] [v1] Sat, 7 Jun 2025 15:50:43 UTC (4,791 KB) [v2] Fri, 3 Apr 2026 14:24:00 UTC (47 KB)