Are Bayesian neural networks intrinsically good at out-of-distribution detection?

The need to avoid confident predictions on unfamiliar data has sparked\ninterest in out-of-distribution (OOD) detection. It is widely assumed that\nBayesian neural networks (BNN) are well suited for this task, as the endowed\nepistemic uncertainty should lead to disagreement in predictions on outliers.\nIn this paper, we question this assumption and provide empirical evidence that\nproper Bayesian inference with common neural network architectures does not\nnecessarily lead to good OOD detection. To circumvent the use of approximate\ninference, we start by studying the infinite-width case, where Bayesian\ninference can be exact considering the corresponding Gaussian process.\nStrikingly, the kernels induced under common architectural choices lead to\nuncertainties that do not reflect the underlying data generating process and\nare therefore unsuited for OOD detection. Finally, we study finite-width\nnetworks using HMC, and observe OOD behavior that is consistent with the\ninfinite-width case. Overall, our study discloses fundamental problems when\nnaively using BNNs for OOD detection and opens interesting avenues for future\nresearch.\n