Cheap Bootstrap for Fast Uncertainty Quantification of Stochastic Gradient Descent
arXiv:2310.11065v2 Announce Type: replace Abstract: Stochastic gradient descent (SGD) or stochastic approximation has been widely used in model training and stochastic optimization. While there is a huge literature on analyzing its convergence, inference on the obtained solutions from SGD has only been recently studied, yet it is important due to the growing need for uncertainty quantification. We investigate two computationally cheap resampling-based methods to construct confidence intervals for SGD solutions. One uses multiple, but few, SGDs in parallel via resampling with replacement from the data, and another operates this in an online fashion. Our methods can be regarded as enhancements of established bootstrap schemes to substantially reduce the computation effort in terms of resampl
View PDF HTML (experimental)
Abstract:Stochastic gradient descent (SGD) or stochastic approximation has been widely used in model training and stochastic optimization. While there is a huge literature on analyzing its convergence, inference on the obtained solutions from SGD has only been recently studied, yet it is important due to the growing need for uncertainty quantification. We investigate two computationally cheap resampling-based methods to construct confidence intervals for SGD solutions. One uses multiple, but few, SGDs in parallel via resampling with replacement from the data, and another operates this in an online fashion. Our methods can be regarded as enhancements of established bootstrap schemes to substantially reduce the computation effort in terms of resampling requirements, while bypassing the intricate mixing conditions in existing batching methods. We achieve these via a recent so-called cheap bootstrap idea and refinement of a Berry-Esseen-type bound for SGD.
Subjects:
Machine Learning (stat.ML); Machine Learning (cs.LG)
Cite as: arXiv:2310.11065 [stat.ML]
(or arXiv:2310.11065v2 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2310.11065
arXiv-issued DOI via DataCite
Journal reference: Journal of Machine Learning Research, 27(25-0008):1-42, 2026
Submission history
From: Zitong Wang [view email] [v1] Tue, 17 Oct 2023 08:18:10 UTC (433 KB) [v2] Tue, 31 Mar 2026 00:09:54 UTC (474 KB)
Sign in to highlight and annotate this article
Conversation starters
Daily AI Digest
Get the top 5 AI stories delivered to your inbox every morning.
More about
modeltrainingannounce
Rocket Close transforms mortgage document processing with Amazon Bedrock and Amazon Textract
Through a strategic partnership with the AWS Generative AI Innovation Center (GenAIIC), Rocket Close developed an intelligent document processing solution that has significantly reduced processing time, making the process 15 times faster. The solution, which uses Amazon Textract for OCR processing and Amazon Bedrock for foundation models (FMs), achieves a strong 90% overall accuracy in document segmentation, classification, and field extraction.
Scaling seismic foundation models on AWS: Distributed training with Amazon SageMaker HyperPod and expanding context windows
This post describes how TGS achieved near-linear scaling for distributed training and expanded context windows for their Vision Transformer-based SFM using Amazon SageMaker HyperPod. This joint solution cut training time from 6 months to just 5 days while enabling analysis of seismic volumes larger than previously possible.
Knowledge Map
Connected Articles — Knowledge Graph
This article is connected to other articles through shared AI topics and tags.
More in Models
Rocket Close transforms mortgage document processing with Amazon Bedrock and Amazon Textract
Through a strategic partnership with the AWS Generative AI Innovation Center (GenAIIC), Rocket Close developed an intelligent document processing solution that has significantly reduced processing time, making the process 15 times faster. The solution, which uses Amazon Textract for OCR processing and Amazon Bedrock for foundation models (FMs), achieves a strong 90% overall accuracy in document segmentation, classification, and field extraction.
Scaling seismic foundation models on AWS: Distributed training with Amazon SageMaker HyperPod and expanding context windows
This post describes how TGS achieved near-linear scaling for distributed training and expanded context windows for their Vision Transformer-based SFM using Amazon SageMaker HyperPod. This joint solution cut training time from 6 months to just 5 days while enabling analysis of seismic volumes larger than previously possible.
Discussion
Sign in to join the discussion
No comments yet — be the first to share your thoughts!