Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping

Using an extended and formalized version of the Q/C map analysis of Poole et\nal. (2016), along with Neural Tangent Kernel theory, we identify the main\npathologies present in deep networks that prevent them from training fast and\ngeneralizing to unseen data, and show how these can be avoided by carefully\ncontrolling the "shape" of the network's initialization-time kernel function.\nWe then develop a method called Deep Kernel Shaping (DKS), which accomplishes\nthis using a combination of precise parameter initialization, activation\nfunction transformations, and small architectural tweaks, all of which preserve\nthe model class. In our experiments we show that DKS enables SGD training of\nresidual networks without normalization layers on Imagenet and CIFAR-10\nclassification tasks at speeds comparable to standard ResNetV2 and Wide-ResNet\nmodels, with only a small decrease in generalization performance. And when\nusing K-FAC as the optimizer, we achieve similar results for networks without\nskip connections. Our results apply for a large variety of activation\nfunctions, including those which traditionally perform very badly, such as the\nlogistic sigmoid. In addition to DKS, we contribute a detailed analysis of skip\nconnections, normalization layers, special activation functions like RELU and\nSELU, and various initialization schemes, explaining their effectiveness as\nalternative (and ultimately incomplete) ways of "shaping" the network's\ninitialization-time kernel.\n