Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training

Normalization techniques have become a basic component in modern\nconvolutional neural networks (ConvNets). In particular, many recent works\ndemonstrate that promoting the orthogonality of the weights helps train deep\nmodels and improve robustness. For ConvNets, most existing methods are based on\npenalizing or normalizing weight matrices derived from concatenating or\nflattening the convolutional kernels. These methods often destroy or ignore the\nbenign convolutional structure of the kernels; therefore, they are often\nexpensive or impractical for deep ConvNets. In contrast, we introduce a simple\nand efficient "Convolutional Normalization" (ConvNorm) method that can fully\nexploit the convolutional structure in the Fourier domain and serve as a simple\nplug-and-play module to be conveniently incorporated into any ConvNets. Our\nmethod is inspired by recent work on preconditioning methods for convolutional\nsparse coding and can effectively promote each layer's channel-wise isometry.\nFurthermore, we show that our ConvNorm can reduce the layerwise spectral norm\nof the weight matrices and hence improve the Lipschitzness of the network,\nleading to easier training and improved robustness for deep ConvNets. Applied\nto classification under noise corruptions and generative adversarial network\n(GAN), we show that the ConvNorm improves the robustness of common ConvNets\nsuch as ResNet and the performance of GAN. We verify our findings via numerical\nexperiments on CIFAR and ImageNet.\n