Summary
This work questions why the linearized Laplace approximation can be effective for uncertainty estimation, whereas the vanilla Laplace approximation can lead to poor performance, such as underfitting. This work argues that this degradation is attributed to the fact that the random function derived from each parameterization of a DNN may not form an invariant function space.
Specifically, since the null space $ \text{ker} (\text{GGN}_{w}) $ is not empty for over-parameterized NNs, sampling the random function containing elements of null space could result in an inconsistent random function. This can lead to under-fitting issues as observed in Laplace approximation.
Therefore, the authors believe that sampling the random function on $ \text{im} (\text{GGN}_{w}) $, i.e., the linearized Laplace, could be more effective than the Laplace method.
Furthermore, for a given weight $w$, they explore how to sample the same function with $ f(w) $, i.e., $ f(g(w)) = f(w) $, by sampling the weight on a specific manifold $ g(w) $. To this end, they employ the concept of quotient space to elaborately define the manifold and diffusion to sample $ w $ from the manifold $ g(w) $.
Strengths
* This work reasonably reveals why the linearized Laplace approximation can be more effective than the standard Laplace approximation. This explanation appears to be novel.
* This work attempts to justify the above reasoning using a mathematical elaboration framework.
Weaknesses
* The lack of explanation of background knowledge, such as quotient space and Riemannian manifold, makes the paper difficult to understand.
* Although it appears effective compared to Laplace and Linearized Laplace, the performance improvement seems marginal when compared to other baselines such as SWAG and Last-layer Laplace.
Questions
* To confirm my understanding, does the Laplace diffusion denote that for a given $w$, the sample functions $ { w_t }$ are obtained by the update rule using $\text{GGN}^{+}$ as described in the SDE on the manifold?
* Does $\text{GGN}^{+}$ mean the positive definite matrix of $\text{GGN}$, which is obtained by applying SVD on $\text{GGN}$ and then using the eigenvectors with positive eigenvalues?
* Is the Laplace diffusion the post-hoc method, meaning that $w$ is first obtained by MAP inference, i.e., $w_{\text{MAP}} = R^{f}_{\mathcal{X}}(w)$, and then $w_t$ is obtained by Laplace diffusion with $\text{GGN}^{+}$?
* Compared to the performance of SWAG described in attached appendix, the performance of the Laplace diffusion does not seem significantly improved.
In this context, for a given $w$, is it important to sample the weight parameter according to the invariant Riemannian manifold?
* Rather, as considering that SWAG focuses on finding good neighborhood of $w$ in training procedure and obtains comparable performance, isn't it more important to focus on how to find good $w$ (for example, $ w_{\text{swa}} $ ) and explore the subspace of $w$? I am just curious about your opinion on this.
Limitations
See above weaknesses and limitations.