Stochastic Gradient Estimation for Artificial Neural Networks

Abstract

In this paper, we consider artificial neural networks (ANN) with processing noise. We derive an infinitesimal perturbation analysis (IPA) and likelihood ratio (LR) gradient estimator for this type of ANN. We show that the IPA estimator, though mathematically equivalent to the backpropagation algorithm, has higher computational complexity than the backpropagation algorithm. To overcome the obstacle posed to IPA and standard stochastic gradient methods by discontinuous activation and loss functions, we introduce in this paper a push-out LR estimator. The push-out LR gradient estimator is computationally more efficient than the backpropagation algorithm, and it avoids differentiating the actual ANN. This allows for discontinuous activation and loss functions to be handled by our LR estimator. We implement the developed stochastic gradient estimation techniques in an example on identifying images of numbers from 0 to 9. The numerical results substantiate the effectiveness of the proposed schemes.