Convergence and Alignment of Gradient Descent with Random Backpropagation Weights

Stochastic gradient descent with backpropagation is the workhorse of\nartificial neural networks. It has long been recognized that backpropagation\nfails to be a biologically plausible algorithm. Fundamentally, it is a\nnon-local procedure -- updating one neuron's synaptic weights requires\nknowledge of synaptic weights or receptive fields of downstream neurons. This\nlimits the use of artificial neural networks as a tool for understanding the\nbiological principles of information processing in the brain. Lillicrap et al.\n(2016) propose a more biologically plausible "feedback alignment" algorithm\nthat uses random and fixed backpropagation weights, and show promising\nsimulations. In this paper we study the mathematical properties of the feedback\nalignment procedure by analyzing convergence and alignment for two-layer\nnetworks under squared error loss. In the overparameterized setting, we prove\nthat the error converges to zero exponentially fast, and also that\nregularization is necessary in order for the parameters to become aligned with\nthe random backpropagation weights. Simulations are given that are consistent\nwith this analysis and suggest further generalizations. These results\ncontribute to our understanding of how biologically plausible algorithms might\ncarry out weight learning in a manner different from Hebbian learning, with\nperformance that is comparable with the full non-local backpropagation\nalgorithm.\n