Less is More: Rethinking Few-Shot Learning and Recurrent Neural Nets

View PDF HTML (experimental)

Abstract:The statistical supervised learning framework assumes an input-output set with a joint probability distribution that is reliably represented by the training dataset. The learner is then required to output a prediction rule learned from the training dataset's input-output pairs. In this work, we provide meaningful insights into the asymptotic equipartition property (AEP) \citep{Shannon:1948} in the context of machine learning, and illuminate some of its potential ramifications for few-shot learning. We provide theoretical guarantees for reliable learning under the information-theoretic AEP, and for the generalization error with respect to the sample size. We then focus on a highly efficient recurrent neural net (RNN) framework and propose a reduced-entropy algorithm for few-shot learning. We also propose a mathematical intuition for the RNN as an approximation of a sparse coding solver. We verify the applicability, robustness, and computational efficiency of the proposed approach with image deblurring and optical coherence tomography (OCT) speckle suppression. Our experimental results demonstrate significant potential for improving learning models' sample efficiency, generalization, and time complexity, that can therefore be leveraged for practical real-time applications.

Comments: Version 3 is focused exclusively on the first part of v1 and v2, correcting minor mathematical errors. The original co-authors have transitioned in separate follow-up works

Subjects:

Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2209.14267 [cs.LG]

(or arXiv:2209.14267v3 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2209.14267

arXiv-issued DOI via DataCite

Submission history

From: Deborah Pereg [view email] [v1] Wed, 28 Sep 2022 17:33:11 UTC (5,801 KB) [v2] Sat, 25 Feb 2023 23:26:13 UTC (9,693 KB) [v3] Sun, 29 Mar 2026 11:58:01 UTC (96 KB)