Summary
The article focuses on the problem of uncertainty quantification in classification.
Calibration provides some guarantees on the estimated class probabilities on average. However, subgroups can still be miscalibrated. The article first aims to characterize these subgroup miscalibrations through proximity levels of the samples. It claims that a classifier, even calibrated, tends to be underconfident on high-proximity samples and overconfident on low-proximity samples. To measure this effect, it defines a proximity-informed ECE. Then, it proposes a recalibration framework based on this proximity-informed measure. Finally, it benchmarks the proposed method on numerous datasets and models.
Strengths
* The problem is well presented and motivated. Figure 1 is pedagogical and helps the comprehension of the problem.
* The idea of characterizing subgroup miscalibrations through proximity is interesting, and refining uncertainty estimates is a good direction.
* The proposed framework is versatile: it can work as a stand-alone or combined with standard calibration techniques. It provides two versions: binning-based and continuous.
* The experimental study is substantial.
* Datasets are large-scale, numerous, and multimodal: ImageNet, Yahoo-Topics, iNaturalist, ImageNet-LT, MultiNLI, ImageNet-C.
* The article studies numerous models, e.g. 504 pre-trained models on ImageNet.
* It compares many standard calibration methods, both scaling-based and histogram-based: temperature scaling, ensemble temperature scaling, parameterized temperature scaling, histogram binning, isotonic regression, and multi-isotonic regression.
* The experimental study provides substantial evidence.
* It reveals proximity bias in most of the 504 pre-trained networks on ImageNet (72% according to a Wilcoxon rank-sum test).
* The proposed method consistently improves over standard calibration methods.
* The time overhead of the method is small, with an increase of 1.17% in inference runtime.
* Completeness of the study: It reveals the proximity bias, proposes a metric to measure it, a recalibration procedure to address it, and substantial experiments showing consistent improvements.
Weaknesses
No major weaknesses.
Questions
* In equation (5) on the confidence score adjustments, why a weight hyperparameter $\lambda \in (0, 1]$ is needed? Don't we want to take $\lambda=1$ to minimize PIECE? The paper talks quickly about regularization. What is the tradeoff? Are results wrong with $\lambda$ too close to 1? Do you take $\lambda=0.5$ everywhere in the experiments? How to choose it?
* Concerning the time overhead introduced by the recalibration methods, isn't it weird that Bin-Mean-Shift has only twice the overhead of isotonic regression (+0.1s vs +0.05s)? Since isotonic regression just needs to apply the mapping function from [0, 1] to [0, 1] to the inferred sample, while Bin-Mean-Shift needs to compute the distance from the new sample to all samples from the calibration set. The same question applies to mapping-based recalibration methods such as histogram binning (+0.03s).
* Could you develop when to use the histogram-based version (section 5.2) and when to use the continuous version (section 5.1) of your method?
---
Typos:
* L148: it should be 80% instead of 85%.
* L162: repeated word Appendix.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
The authors discuss the following limitations:
* The proposed recalibration technique needs to maintain the calibration set during inference to compute the proximity of the new points.
* Maintaining the calibration set for inference may challenge the method when memory is limited.
* Focus limited to the closed-set multi-class classification problem.