Summary
The paper questions the reliance of probabilistic classifiers on the confidence score alone for measuring reliability. This is an important question that has not been asked that rigorously in the machine learning literature. In particular, there are two notions of uncertainty: aleatoric and epistemic. A widespread consensus is to consider the softmax applied output of a neural network as the confidence of the classifier. However, it is not clear what kind of uncertainty that score captures. Literature in proper scoring rule would suggest that it measures the posterior probability of the input falling into some class, which is inherently an aleatoric notion of uncertainty. However, practitioners and researchers in the community also hope that such a score would be measure of reliability of rare / out-of-distribution / atypical samples, which is more of an epistemic nature of uncertainty. Therefore, there is some sort of conflation is happening on the nature of this score. A sample could be low confidence because it is inherently ambiguous (aleatoric) or is not well represented in the training distribution (epistemic). The paper opens up with this question, and gives arguments to quantify *atypicality* of the sample as well for more reliability. With a simple measure to quantify atypicality, it shows that there is a a correspondence between how atypical a sample is and how mis-calibrated it is. It then studies atypicality aware recalibration. There is a sufficient empirical evaluation.
Strengths
1. One of the major strengths of this paper is that it raises a very relevant question. Interpretation of the confidence outputted by the classifier has a tendency to suffer from serious conflation issues where it is not clear what it captures. The paper argues for a different notion that can settle this issue (or at least start interesting discussions in the community). I believe this is highly valuable.
2. While the operationalisation of the proposed atypicality score is simple, the scoring criteria is already good enough to support the arguments made in the paper in a satisfactory way.
3. Thorough empirical evaluation is done on different settings ranging from balanced supervised classification, imbalanced classification , Language modelling, recalibration, and conformal prediction uncertainty sets on a range of datasets.
Weaknesses
1. There are no major glaring weaknesses of the paper. Except that maybe it underplays itself, as in my mind, it puts interesting and insightful commentary on the nature of softmax output (or sigmoid for Binary classification) of the classifier. There is a great debate on what that is, see Appendix B.2 in [1]. The paper could be even more impactful if these discussions are taken into consideration.
2. Section 3.2: Although some theoretical results are presented that shows correspondence between atypicality and overconfidence in a simple model, it is not clear how these results apply to the other things in the paper. It would be good to draw other insights from it. Do these results provide any insight (or intuition) on why atypical examples would be mis-calibrated? Is this due to estimation errors?
Personally, I thought about this question myself. And to think of it, it does make sense intuitively why mis-calibration might be an issue for atypical examples. Consider Binary classification, and assume that one is perfectly able to model $\eta(x) = \mathbb{P}(Y=1\vert X=x)$ where the evaluation of $\mathbb{P}$ happens with respect to the training data distribution $P$. Now for a sample $x_{new}$ that is not well-represented in $P$, it is not unsurprising to see that $\mathbb{P}(Y=1 \vert X=x_{new})$ would be low. However, there is no reason to expect that it would be calibrated. Could authors comment on or augment this intuition?
[1] Tim Pearce et al. Understanding Softmax Confidence and Uncertainty. (2021)
Questions
Some questions follow:
1. (Section 4.2): I'm not sure I follow the intuition behind Equation 2. $\psi$ is a function of input atypicality. What is the nature of $\psi$? Is it monotonic, as in it preserves the atypicality of the input. If I assume so, then for a typical input, let's say the atypicality score is 0, in this case $\hat{\mathbb{P}}_{\text{ARR}}(Y\vert X) = \frac{\exp\{S_Y\}}{Z(X)}$, which is not intuitive. I'm sure this is my misunderstanding, so would appreciate the clarification. Specifically, the nature of $\psi$ and $\phi$.
2. (Line 222): The paper shows that atypicality-aware recalibration improves the overall accuracy in imbalanced supervised classification. In my mind, this is problematic, no? Recalibration techniques should preserve the nature of classifier, is called the sharpness / refinement property of calibration maps. Furthermore, ECE roughly translates to difference in average confidence and average accuracy. And the high ECE in atypicality is the result of lower accuracy but higher confidence. One could improve on ECE by compensating for accuracy instead of the confidence, which in my opinion is not the goal of calibration. I would love to hear from the authors on this.
3. (Section 4.3): I understand that atypicality aware recalibration is useful in fairness methods. Although it is an interesting case study and not the major focus of the paper, could authors comment (and compare) on the notion of multicalibration in the literature. I think atypicality aware recalibration has the same spirit as multicalibration in fairness literature.
4. (Section 5, Line 266): For conformal prediction, the authors write "group points according to their confidence and atypicality quantiles...", do the authors use some function to club the confidence score and the atypicality score into a single score? Or some multiple testing procedure is used to accommodate both the score?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.