The skip-connections used in residual networks have become a standard\narchitecture choice in deep learning due to the increased training stability\nand generalization performance with this architecture, although there has been\nlimited theoretical understanding for this improvement. In this work, we\nanalyze overparameterized deep residual networks trained by gradient descent\nfollowing random initialization, and demonstrate that (i) the class of networks\nlearned by gradient descent constitutes a small subset of the entire neural\nnetwork function class, and (ii) this subclass of networks is sufficiently\nlarge to guarantee small training error. By showing (i) we are able to\ndemonstrate that deep residual networks trained with gradient descent have a\nsmall generalization gap between training and test error, and together with\n(ii) this guarantees that the test error will be small. Our optimization and\ngeneralization guarantees require overparameterization that is only logarithmic\nin the depth of the network, while all known generalization bounds for deep\nnon-residual networks have overparameterization requirements that are at least\npolynomial in the depth. This provides an explanation for why residual networks\nare preferable to non-residual ones.\n