Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks

The selection of initial parameter values for gradient-based optimization of\ndeep neural networks is one of the most impactful hyperparameter choices in\ndeep learning systems, affecting both convergence times and model performance.\nYet despite significant empirical and theoretical analysis, relatively little\nhas been proved about the concrete effects of different initialization schemes.\nIn this work, we analyze the effect of initialization in deep linear networks,\nand provide for the first time a rigorous proof that drawing the initial\nweights from the orthogonal group speeds up convergence relative to the\nstandard Gaussian initialization with iid weights. We show that for deep\nnetworks, the width needed for efficient convergence to a global minimum with\northogonal initializations is independent of the depth, whereas the width\nneeded for efficient convergence with Gaussian initializations scales linearly\nin the depth. Our results demonstrate how the benefits of a good initialization\ncan persist throughout learning, suggesting an explanation for the recent\nempirical successes found by initializing very deep non-linear networks\naccording to the principle of dynamical isometry.\n