Depthwise Separable Convolutions Allow for Fast and Memory-Efficient Spectral Normalization

An increasing number of models require the control of the spectral norm of\nconvolutional layers of a neural network. While there is an abundance of\nmethods for estimating and enforcing upper bounds on those during training,\nthey are typically costly in either memory or time. In this work, we introduce\na very simple method for spectral normalization of depthwise separable\nconvolutions, which introduces negligible computational and memory overhead. We\ndemonstrate the effectiveness of our method on image classification tasks using\nstandard architectures like MobileNetV2.\n

cs.LG cs.CV stat.ML Neural Networks and Applications Image and Signal Denoising Methods Human Pose and Action Recognition

Paper

Similar papers

Convolutional Normalization: Improving Deep Convolutional Network Robustness and TrainingSheng Liu, Xiao Li, Yuexiang Zhai et al.2021 · arXiv (Cornell University) · 29 citations In Library

Network Decoupling: From Regular to Depthwise Separable ConvolutionsJianbo Guo, Yuxi Li, Weiyao Lin et al.2018 · British Machine Vision Conference · 100 citations In Library

Fast Approximate Spectral Normalization for Robust Deep Neural NetworksZhixin Pan, Prabhat Mishra2021 · arXiv (Cornell University) · 1 citation In Library

Depth-wise Decomposition for Accelerating Separable Convolutions in Efficient Convolutional Neural NetworksYihui He, Jianing Qian, Cindy X. Le et al.2023 · Advances in Artificial Intelligence and Machine Learning In Library

Depth-wise Decomposition for Accelerating Separable Convolutions in Efficient Convolutional Neural NetworksYihui He, Jianing Qian, Jianren Wang2019 · Advances in Artificial Intelligence and Machine Learning · 14 citations In Library

Fantastic Four: Differentiable and Efficient Bounds on Singular Values of Convolution LayersSahil Singla, Soheil Feizi2021 · International Conference on Learning Representations In Library

Accelerating Spectral Normalization for Enhancing Robustness of Deep Neural NetworksZhixin Pan, Prabhat Mishra2021 · IEEE Computer Society Annual Symposium on VLSI · 1 citation In Library

Tight and Efficient Upper Bound on Spectral Norm of Convolutional LayersEkaterina Grishina, Mikhail Gorbunov, Maxim Rakhuba2024 · European Conference on Computer Vision · 8 citations In Library

An Energy-Efficient Convolution Unit for Depthwise Separable Convolutional Neural NetworksYi Sheng Chong, Wang Ling Goh, Yew-Soon Ong et al.2021 · International Symposium on Circuits and Systems · 8 citations In Library

Generalized Depthwise-Separable Convolutions for Adversarially Robust and Efficient Neural NetworksHassan Dbouk, Naresh R. Shanbhag2021 · arXiv (Cornell University) · 8 citations In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal