Summary
The paper “Minimizing Hyperspherical Energy for Diverse Deep Ensembling” explores the use of Centered Kernel Alignment (CKA) and Minimization of Hyperspherical Energy (MHE) in Bayesian deep learning to enhance the diversity of ensemble models and parameters generated by hypernetworks. By incorporating these techniques, the authors aim to improve uncertainty quantification in both synthetic and real-world tasks, which is quantified by measuring OOD detection performance and calibration. The key contributions include proposing CKA as an optimization objective and utilizing MHE to address the diminishing gradient issue, leading to more stable training and better performance in uncertainty estimation.
Strengths
- **Original:** The paper introduces a novel approach by combining CKA and MHE to enhance the diversity of deep learning ensembles and hypernetworks, which is a fresh perspective in Bayesian deep learning.
- **Detailed experiments:** The experimental results are comprehensive and demonstrate significant improvements in uncertainty quantification across various tasks, showing the practical effectiveness of the proposed methods.
- **Well-written:** The paper is well-structured, with clear explanations of the methods and thorough discussions of the results. The figures and tables effectively illustrate the performance improvements.
Weaknesses
- The comparison of the approach in terms of OOD detection performance is slightly biased due to leveraging of generated OOD samples. Here in order to differentiate the contribution of the method from the incorporation of additional information a further experiments comparing against DDU would be warranted. In particular one could imagine adding a term to the loss which encourages low likelihood of the fitted GMM on OOD samples.
- The comparisons in Figure 2 paint a slightly overly optimistic picture for the OOD HE approaches. In 2 dimensions the problem becomes quite trivial given negative / OOD samples. The difficulty of generating enough samples to cover the OOD volume becomes exponentially more difficult with increasing dimensionality of the input space and is especially easy when the diversity of the data is limited.
- (Related to prior point) the datasets used to show the efficacy of the OOD approach are rather simple and small such that sample diversity is limited. Running one experiment on a larger dataset like imagenet would be very beneficial to show that the method can also work in the context of highly diverse input distributions and high dimensionality of the input space.
- There seems to be a mistake in table 2 row Ensemble + HE, column PE
- I recommend renaming Fig1 d and e to Cossim evaluation and HE evaluation as using objective gives the impression that the data shown is from models trained using these objectives.
- As the work points out that applying the method to larger datasets would in principle be possible, this might be a good additional comparison to include. Many comparison approaches will have issues in this case, but comparison to other ensembling based approaches would be feasible.
Questions
- When running the LeNet comparisons agains DDU, was the model trained with spectral normalization (a requirement for DDU to function well) and how was the regularization strength determined?
- Why was the normalization-free variant of LeNet used for some methods whereas the standard variant for the DDU baseline?
- Page 9, line 289: Was the GMM fitted to the features of the pre-classification layer. I.e. to the features used by the last linear layer?
Limitations
The work correctly points out that the small size of the datasets is one of the main limitations. Generally, seeing the efficacy of the approach in a larger scale training setup would significantly improve the paper.