On Convergence and Generalization of Dropout Training

We study dropout in two-layer neural networks with rectified linear unit (ReLU) activations. Under mild overparametrization and assuming that the limiting kernel can separate the data distribution with a positive margin, we show that dropout training with logistic loss achieves $\epsilon$-suboptimality in test error in $O(1/\epsilon)$ iterations.

cs.LG stat.ML Neural Networks and Applications Stochastic Gradient Optimization Techniques Machine Learning and ELM Computer Science Mathematics

View graph Source

Paper

References (48)

01High-Dimensional Probability: An Introduction with Applications in Data ScienceO. Papaspiliopoulos2020 · Quantitative finance (Print) · 4.1k citations

02Can Implicit Bias Explain Generalization? Stochastic Convex Optimization as a Case StudyAssaf Dauber, M. Feder, Tomer Koren et al.2020 · Neural Information Processing Systems · 26 citations In Library

03Dropout: Explicit Forms and Capacity ControlR. Arora, P. Bartlett, Poorya Mianjy et al.2020 · International Conference on Machine Learning · 44 citations In Library

04The Implicit and Explicit Regularization Effects of DropoutColin Wei, S. Kakade, Tengyu Ma2020 · International Conference on Machine Learning · 131 citations In Library

05Almost Sure Convergence of Dropout Algorithms for Neural NetworksAlbert Senen-Cerda, J. Sanders2020 · Journal of machine learning research · 11 citations In Library

06Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networksZiwei Ji, Matus Telgarsky2019 · International Conference on Learning Representations · 197 citations In Library

07Quadratic Suffices for Over-parametrization via Matrix Chernoff BoundZhao Song, Xin Yang2019 · arXiv.org · 97 citations In Library

08Learning Sparse Networks Using Targeted DropoutAidan N. Gomez, Ivan Zhang, Kevin Swersky et al.2019 · arXiv.org · 103 citations In Library

09Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural NetworksYuan Cao, Quanquan Gu2019 · Neural Information Processing Systems · 434 citations In Library

10Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization GuaranteeWei Hu, Zhiyuan Li, Dingli Yu2019 · International Conference on Learning Representations · 144 citations

11On Dropout and Nuclear Norm RegularizationPoorya Mianjy, R. Arora2019 · International Conference on Machine Learning · 25 citations In Library

12Refined Generalization Analysis of Gradient Descent for Over-parameterized Two-layer Neural Networks with Smooth Activations on Classification ProblemsAtsushi Nitanda, Taiji Suzuki2019 · arXiv.org · 16 citations

Scroll for more · 36 remaining

Similar papers

Almost Sure Convergence of Dropout Algorithms for Neural NetworksAlbert Senen–Cerda, Jaron Sanders2020 · arXiv (Cornell University) · 11 citations In Library

Implicit regularization of dropoutZhongwang Zhang, Zhi‐Qin John Xu2022 · arXiv (Cornell University) · 62 citations In Library

Implicit Regularization of DropoutZhongwang Zhang, Zhi‐Qin John Xu2024 · IEEE Transactions on Pattern Analysis and Machine Intelligence In Library

Asymptotic convergence rate of Dropout on shallow linear neural networksAlbert Senen–Cerda, Jaron Sanders2020 · arXiv (Cornell University) · 9 citations In Library

On the Inductive Bias of DropoutD. Helmbold, Philip M. Long2014 · Journal of machine learning research · 74 citations In Library

On the Convergence of Shallow Neural Network Training with Randomly Masked NeuronsFangshuo Liao, Anastasios Kyrillidis2021 · arXiv (Cornell University) · 16 citations In Library

Asymptotic Convergence Rate of Dropout on Shallow Linear Neural NetworksAlbert Senen–Cerda, Jaron Sanders2022 · Proceedings of the ACM on Measurement and Analysis of Computing Systems In Library

Asymptotic Convergence Rate of Dropout on Shallow Linear Neural NetworksAlbert Senen–Cerda, Jaron Sanders2022 · ACM SIGMETRICS Performance Evaluation Review In Library

Asymptotic Convergence Rate of Dropout on Shallow Linear Neural NetworksAlbert Senen–Cerda, Jaron Sanders2022 In Library

An empirical analysis of dropout in piecewise linear networksDavid Warde-Farley, I. Goodfellow, Aaron C. Courville et al.2013 · International Conference on Learning Representations · 111 citations In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal