Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networks

Training neural networks with binary weights and activations is a challenging\nproblem due to the lack of gradients and difficulty of optimization over\ndiscrete weights. Many successful experimental results have been achieved with\nempirical straight-through (ST) approaches, proposing a variety of ad-hoc rules\nfor propagating gradients through non-differentiable activations and updating\ndiscrete weights. At the same time, ST methods can be truly derived as\nestimators in the stochastic binary network (SBN) model with Bernoulli weights.\nWe advance these derivations to a more complete and systematic study. We\nanalyze properties, estimation accuracy, obtain different forms of correct ST\nestimators for activations and weights, explain existing empirical approaches\nand their shortcomings, explain how latent weights arise from the mirror\ndescent method when optimizing over probabilities. This allows to reintroduce\nST methods, long known empirically, as sound approximations, apply them with\nclarity and develop further improvements.\n