Sampling-free Variational Inference for Neural Networks with Multiplicative Activation Noise

To adopt neural networks in safety critical domains, knowing whether we can\ntrust their predictions is crucial. Bayesian neural networks (BNNs) provide\nuncertainty estimates by averaging predictions with respect to the posterior\nweight distribution. Variational inference methods for BNNs approximate the\nintractable weight posterior with a tractable distribution, yet mostly rely on\nsampling from the variational distribution during training and inference.\nRecent sampling-free approaches offer an alternative, but incur a significant\nparameter overhead. We here propose a more efficient parameterization of the\nposterior approximation for sampling-free variational inference that relies on\nthe distribution induced by multiplicative Gaussian activation noise. This\nallows us to combine parameter efficiency with the benefits of sampling-free\nvariational inference. Our approach yields competitive results for standard\nregression problems and scales well to large-scale image classification tasks\nincluding ImageNet.\n