One Simple Trick to Fix Your Bayesian Neural Network

One of the most popular estimation methods in Bayesian neural networks (BNN)[Blundell et al., 2015] is mean-field variational inference (MFVI)[Blei et al., 2017]. In this work, we show that neural networks with ReLU[Fukushima, 1975] activation function induce posteriors, that are hard to fit with MFVI. We provide a theoretical justification for this phenomenon, study it empirically, and report the results of a series of experiments to investigate the effect of activation function on the calibration of BNNs. We find that using Leaky ReLU activations[Maas et al., 2013] leads to more Gaussian-like weight posteriors and achieves a lower expected calibration error (ECE)[Guo et al., 2017] than its ReLU-based counterpart.