How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networks

Response to Reviewer TapJ

Thank you very much for your constructive feedback. Please find our responses below. We hope they are satisfactory and you will consider raising your score. >**W1:** Insufficient comparison to post-training methods. (...) there are several similarly simple (in terms of parameters) approaches introduced in [1], (...) which are not compared to in the main results of the paper. Thank you for your suggestion. We included a new Appendix H with a comparison to other existing tunable methods, including the main method proposed by Balanya et al. (2023) (we assume this is the [1] you are referring to). As can be seen from the table below, MaxLogit-pNorm showed superior performance to all of these methods, while requiring much less hyperparameter tuning. | Method | APG-NAURC | |-|-| | ETS (Zhang et al., 2020) | 0.05776 | | BK (Boursinos & Koutsoukos, 2022) | 0.02504 | | HTS (Balanya et al., 2023) | 0.06286 | | MaxLogit-pNorm | **0.06837** | >**W2:** Discussion of related work on calibration. (...) I think the work would benefit from discussion of modifications to TS/related strategies with more justification for why comparing $p$-norm normalization with vanilla TS is sufficient for making their points. Thank you for this suggestion. We have included in the new Appendix H a review of the methods mentioned in the above table as well as a discussion of post-hoc calibration methods with a large number of tunable parameters and a justification for not including them in our evaluation. >**Q1:** In Section 3.2.2, the authors write: "a useful property of MSP-TS (but not MSP-TS-NLL) is that it can never have worse performance than the MSP baseline"; I do not see why this is true. As long as we do TS on held-out calibration/validation data, it is certainly still possible that any metric could be worse on test data. The reviewer is correct; thanks for pointing this out. We have fixed the sentence as: "A useful property of MSP-TS (but not MSP-TS-NLL) is that, *in the infinite-sample setting,* it can never have a worse performance than the MSP baseline, as long as T = 1 is included in the search space." >**Q2:** What is the best choice of $p$ found by tuning $p$-norm? It would be useful to include this in the main discussion, and whether there is actually some significant benefit to tuning $p$ instead of just using $p$=2. For MaxLogit-pNorm, the best choice of $p$ for each model is shown in the legend of Fig. 3, where we can see it is typically between 4 and 6. It is also given in the appendix tables with the full results for MSP-pNorm and MaxLogit-pNorm. We have now included a new Appendix G with a comparison between tunable and fixed $p$ for these two methods and included a pointer to these results in Section 4.1. As can be seen in the table below, there is a significant benefit of tuning $p$ compared to just using $p=2$. Note that the statistical significance of these benefits is discussed in the response to the next question. |$p$|MaxLogit-pNorm|MSP-pNorm| |-|-|-| |0|0.00000|0.05776| |1|0.00197|0.06030| |2|0.01471|0.06501| |3|0.05039|0.06714| |4|0.06416|0.06798| |5|0.06778|0.06787| |6|0.06785|0.06741| |7|0.06654|0.06703| |Tunable|0.06837|0.06781| >**Q3:** In Table 3, it would be useful to include error bounds (like 1 std range) since you are reporting averages across all models here to better get a sense of the improvement. Please note that we are reporting averages across models simply as a way to summarize performance, but it would be potentially misleading to report error bars for these averages, as the choice of a model is not a random variable, but rather a specific experiment scenario. In our data efficiency experiments (Figures 10 and 11 in the new version), we had already shown 10-90 percentile regions, which are obtained by randomly varying the validation-test split, for a specific model ResNet50. We have now included a new Appendix I where we perform a similar analysis for all models. Figures 12 and 13 (shown below) show +-1 standard deviation around the mean, with models sorted by NAURC gain over MSP for MaxLogit-pNorm (similarly to Fig. 2). We can see that, when the validation set has 5000 samples, the gains of MaxLogit-pNorm are statistically significant compared to MSP-TS and MSP-pNorm with fixed $p=2$, but not compared to MSP-pNorm with tunable $p$, which was already suggested by Fig. 2. However, when the validation set is reduced to 1000 samples, MaxLogit-pNorm outperforms all three methods, which was already suggested by the data efficiency experiments. Moreover, we can see that MaxLogit-pNorm is the most stable among these methods in terms of variance of gains. [Click here to see comparison plots with error bounds](https://postimg.cc/5HZTFxgm) We hope we have addressed all of the reviewer’s concerns satisfactorily and would be happy to answer any other questions.

Response to Reviewer 3CxY

Thank you very much for your constructive feedback. Please find our responses below. > I’m not quite motivated from Figure 1. What’s the reason that the left figure is bad in terms of constructing a guaranteed selective classifier? Probably, adding some context would be useful in motivating readers. We agree that it would be beneficial to add more context to Fig 1 and we thank the reviewer for pointing this out. We will replace the caption of Fig 1 with: *Figure 1: A comparison of RC curves made by three models selected in (Galil et al., 2023), including examples of highest (ViT-L/16-384) and lowest (EfficientNet-V2-XL) AUROC. An RC curve shows the tradeoff between risk (in this case, error rate) and coverage. The initial risk for any classifier is found at the 100% coverage point, where all predictions are accepted. Normally, the risk can be reduced by reducing coverage (which is done by increasing the selection threshold); for instance, a 2% error rate can be obtained at 36.2% coverage for the ViT-B/32-224-SAM model and at 61.9% coverage for the ViT-L/16-38 model. However, for the EfficientNet-V2-XL model, this error rate is not achievable at any coverage, since its RC curve is lower bounded by 5% risk. Moreover, this RC curve is actually non-monotonic, with an increasing risk as coverage is reduced, for low coverage. Fortunately, this apparent pathology in EfficientNet-V2-XL completely disappears after a simple post-hoc optimization of its confidence estimator, resulting in significantly improved selective classification performance. In particular, a 2% error rate can then be achieved at 55.3% coverage.* We hope this explanation clarifies why the red curve on the left figure is not desirable in terms of constructing a selective classifier with a low risk. > What’s the benefits of conclusion of this paper in terms of the performance of the final guaranteed risk controlled selective classifier by Geifman & El-Yaniv (2017)? The confidence estimator with a good NAURC should demonstrate its benefits along with a selective classifier (in terms of achieving a desired risk level) for in-distribution and out-of-distribution experiments. We agree that an important (and tangible) way to evaluate the performance of a selective classifier is to compute the maximum coverage allowed such that the corresponding selective risk is below some target level (this is essentially the definition of the SAC metric). But note that this information, for all risk levels, can be directly obtained from the RC curve: we choose a desired risk level on the y axis and then look for the rightmost point that crosses this level. Thus, the RC curve provides the evaluation suggested by the reviewer. In this context, some of the conclusions of our paper are: 1. Before post-hoc optimization, many models show pathological RC curves, where at some point the risk starts to increase as coverage decreases. But, after optimization, all RC curves start to look very similar to Fig 1, with the risk going to zero as coverage decreases. Further examples of RC curves can be found in Fig 5 for ImageNet (and Figs 8 and 9 for other datasets), but rather than plotting 84 similar-looking RC curves, we show in Fig 3(b) that, after post-hoc optimization, the AURC becomes extremely (Spearman-)correlated to the classifier accuracy, suggesting that better selective performance (for any risk level) can be obtained simply by choosing a model with better accuracy. 2. In order to give a more concrete evaluation in a distribution shift scenario, we show in Table 3 the maximum coverage allowed such that the accuracy is the same as the original accuracy of the model without distribution shift. We can see that, while coverage always drops with distribution shift (the RC curve moves up), it drops much less after post-hoc optimization is applied. This is not obvious a priori, since the confidence estimator is optimized with in-distribution data, so optimization could potentially be harmful to out-of-distribution generalization, but it is actually beneficial. Motivated by the reviewer's comment, in order to make point 1 above clearer, we will include Figure 7 in the Appendix D.5 showing the SAC metric for a target 98% accuracy for all 84 models on ImageNet, before and after post-hoc optimization. This figure shows that most models achieve significant coverage gains with post-hoc optimization. Moreover, for several models, achieving this target accuracy is virtually impossible without post-hoc optimization (except with coverage very close to 0%). Thus, post-hoc optimization is highly beneficial in achieving a desired target risk. For convenience, a few selected points are shown below (SAC = coverage for 98% accuracy): |Model|Accuracy[%]|SAC-Baseline[%]|SAC-Optimized[%]| |-|-|-|-| |VGG16|71.56|29.79|29.79| |ResNet50|80.82|0.17|45.58| |WideResNet50-2|81.55|0.14|47.78| |ConvNext-Base|84.02|0.22|53.14| |EfficientNetV2-XL|85.55|0.03|55.30| |Vit-h-14|88.48|61.34|65.26|

Response to Reviewer aDAg (1)

Thank you very much for your constructive feedback. Please find our responses below. We hope they are satisfactory and you will consider raising your score. >**W1/Q3:** Why is learnable p-norm better than L2-norm? Why is a different norm required for different datasets? How significant is the benefit of this learnability? I think an ablation study can be better to support the use of learnable p. Thank you for this suggestion. We have added a new Appendix G with an ablation study regarding the choice of p. As shown in the table below, for MaxLogit-pNorm, a tunable p achieves much higher gains than a fixed p=2. For MSP-pNorm, the benefits (of using a tunable p rather than a fixed p=2) are smaller, although still statistically significant (see the Appendices F and I of the new version). However, it should be noted that, differently from MaxLogit, using MSP with p-norm normalization always requires the tuning of the temperature (otherwise the performance deteriorates). This additional tuning, besides being less computationally efficient (since more values need to be evaluated), causes worse data efficiency, as shown in Appendix F (note that we added the data efficiency curve for MSP-pNorm with fixed p=2). |$p$|MaxLogit-pNorm|MSP-pNorm| |-|-|-| |0|0.00000|0.05776| |1|0.00197|0.06030| |2|0.01471|0.06501| |3|0.05039|0.06714| |4|0.06416|0.06798| |5|0.06778|0.06787| |6|0.06785|0.06741| |7|0.06654|0.06703| |Tunable|0.06837|0.06781| >**W2:** The referenced paper introduces logit normalization during training-time, whereas this study applies it as a post-hoc measure. The amount of novelty this aspect brings is somewhat limited. Concerning specifically logit normalization, we would like to emphasize the following points about the novelty of this aspect of our work: * (Wei et al., 2022) proposed normalization in the context of calibration and OOD detection. We propose logit normalization to improve selective classification performance. These are different tasks whose performance is not aligned, e.g., improving calibration may harm selective classification and vice-versa. * (Wei et al., 2022) only used logit normalization during training time; during inference, they still used the raw MSP as confidence estimator, without any normalization. It is currently unknown if this would bring any benefits to selective classification. In contrast, we propose to use it during inference, which can be easily applied on top of any trained model without requiring modifications to its training regime. * (Wei et al., 2022) used only the L2 norm, while we propose using a general p-norm. This extension is not only significant, as shown in the new Appendices F, G, and I, but also theoretically grounded, as shown in the new Appendix C. To the best of our knowledge, applying logit p-norm normalization had not been proposed before in any context. * (Wei et al., 2022) used what we would call “MSP-pNorm with fixed p=2”, which still requires tuning a temperature hyperparameter due to the use of a softmax function. In contrast, we have shown that applying logit p-norm normalization followed by MaxLogit not only achieves a better performance, but also avoids the temperature tuning, resulting in better stability and data efficiency (see the new Appendices F and I). * There is a gap in the literature with respect to the choice of the best post-hoc confidence estimator for selective classification, with many previous works using the MSP as baseline. Even if MSP-pNorm with fixed p=2 turned out to be the best method in our benchmark, performing such a benchmark is (we believe) a significant contribution to this literature. Please note that, besides these, there are also several other original aspects in our work: * We solve an open problem reported in (Galil et al., 2023), which is to fix the anomaly in selective classification observed in many classifiers. In particular, we show that this is not an intrinsic limitation of these classifiers, but simply a consequence of using the MSP; when a better confidence estimator is used (which can be easily found with a few evaluations on hold-out validation data), the anomaly disappears. * We investigate the robustness to distribution shift, both of the proposed post-hoc fix as well as of selective classification itself (after the post-hoc fix). As far as we know, our paper is the first to explore this aspect.

Response to Reviewer aDAg (2)

>**W3/Q9:** There's inconsistency in the performance metrics used in the experiments, with initial methods based on NAURC and subsequent ones relying on SAC, without offering a comparative analysis across all metrics in the appendix. If time permits, it would be better to include a result for NAURC in Table 4 for consistent comparison. Thank you for this observation and suggestion. We have added the NAURC results for all models in Section 4.3 (see the new Figure 4). Moreover, we also added to a new Appendix D.5 the SAC results for all models, which reinforces our conclusions of Section 4.2. The reason to use different metrics is the following: when comparing confidence estimators across different models and/or test datasets (which is the main goal of this paper), the NAURC is an appropriate metric as argued in Section 3.2. However, after the best confidence estimator is chosen for each model and we wish to compare models, then other metrics are potentially more meaningful; that is why we include AURC (and now SAC) results in Section 4.2. Moreover, the SAC is arguably a more tangible metric, that is why we wanted to include it towards the end of our paper, to give a more concrete illustration of the impact of selective classification (note that we significantly rewrote Section 4.3 to make this all clear). Nevertheless, we should emphasize that, in all our experiments, the results are consistent across all metrics. >**W4:** The paper could be significantly improved by providing more intuition/motivation and explaining the rationale behind methodological choices. For instance, the reasons why p-norm would perform effectively in selective classification are not well-articulated. The paper references Wei et al. (2022) but does not provide adequate context for readers less familiar with this work, compromising the clarity of the research. We have included a new Appendix C (and a pointer to it on Section 3.1.2) containing appropriate context on logit normalization during training as proposed by Wei et al. (2022), as well as a detailed explanation of our intuition and theoretical justification for p-norm normalization as a post-training technique. In summary, while our original intuition was that using logit normalization post-training would have a similar behavior to using it during training, our experimental results (Appendix J of the new version) actually disproved this hypothesis. We now have been able to articulate a partial explanation, showing that logit p-norm normalization (with a tunable p) is closely related to temperature scaling (where p acts as 1/T). We hope the reviewer is satisfied with our new material. >**W5:** The research's significance is limited, as it doesn't introduce considerable challenges beyond what is already known from the application of logit normalization to selective classification. Furthermore, intriguing findings, like those depicted in Figure 11, are not sufficiently highlighted or explained in the main text, representing a missed opportunity to underscore potential novel insights. The reasons behind the inability of high-confidence models to enhance selective classification remain unexplored, leaving gaps in understanding. As explained in our response to W2, our paper contains considerable novelty beyond what is already known, and not only in aspects related to logit normalization, but also in selective classification in general and its robustness to distribution shift. Regarding the intriguing findings mentioned by the reviewer, we were not able to elaborate further in the main text due to lack of space, but we have now included additional insights in Appendix C. Nevertheless, the fact that our paper poses a new open problem should not, in our opinion, constitute grounds for rejection; on the contrary, we believe this difficult question is intriguing enough to merit reporting, and we hope to inspire other researchers to investigate it in the future. >**Q1:** What is the intuition behind using p-normalization? Why is logit information helpful in distinguishing between correctly classified and incorrectly classified samples? Using logit information is natural since it is readily available for any softmax-based classifier and the widely used MSP is itself a function of the logits. For this reason many previous works have considered functions of the logits to try to improve upon this baseline, across many related problems such as misclassification detection, calibration and OOD detection. Using logit information, possibly with a few tunable parameters, is also the simplest possible approach, so in this paper our goal was to extensively benchmark these techniques in order to establish a strong baseline for future research. Regarding the former question, we have included a new Appendix C with the intuition behind p-norm normalization, as explained in our response to W4.

Response to Reviewer aDAg (3)

>**Q2:** In instances where logit normalization proves ineffective, could the issue stem from the original model being under-calibrated, blurring the distinction between correctly and incorrectly classified samples due to a minimal confidence gap? My guess of why the logit normalization works is that: it is actually making the correct sample have much higher confidence while incorrect samples have slightly lower confidence, and therefore the confidence can distinguish them well. If this is the case, would it be possible to consider the corresponding ECE (compared to the histogram) to see if under-confident models are more easily improved, while over-confident models are not? We believe the reviewer means “In instances where logit normalization proves **effective**” and that “under-calibrated” means the same as “underconfident”. While we would not have time now to plot calibration curves (not simply ECE as the issue here is underconfidence), the results mentioned by the reviewer (Appendix J of the new version) already confirm this hypothesis: since all of these ImageNet models have relatively high accuracy, whenever their MSP histogram is concentrated on relatively low values, it follows that the model is underconfident. This happens essentially for all models where logit normalization (as well as temperature scaling) is effective, i.e., for the models which have relatively poor performance on selective classification using the MSP. Whether these models are underconfident due to insufficient training or some other reason, it is a question that we hope to address in our future work as well as inspire other researchers to investigate. >**Q4:** What loss objective was used in the training of 'p' and 't'? Was it AURC or normalized AURC? We used AURC as mentioned in Section 3.1.2. However, note that, when the classifier is fixed and we are evaluating some parameter of a confidence estimator, NAURC is a strictly monotonic function of AURC, since it is a simple min-max scaling (see Section 3.2.2), so optimizing AURC is equivalent to optimizing NAURC. >**Q5:** How was MSP FALLBACK considered in the experiments? (...) For a dataset, what criteria should we use to decide whether to use MSP or logit normalization? The MSP fallback was only used to compute APG and whenever explicitly mentioned in the caption of a figure. Assuming the original dataset is split into training (used for training the classifier and never used in this paper), validation and test sets, we should use the validation set to tune the hyperparameters of a confidence estimator. The MSP fallback is then interpreted as a binary hyperparameter, or more generally as an additional configuration to our hyperparameter search grid, e.g., {p=4, p=5, p=6, p=7, MSP}, as illustrated in Figure 3. >**Q6:** Is the third graph in Figure 3 also at 0.3491? It seems to have a high correlation. Indeed there was a plot error, thank you for pointing this out. The correct value for the correlation in this graph is -0.9019. >**Q7:** Can you provide a more detailed explanation of Figure 3? What kind of message can we read from different optimal values of 'p'? Are models with different 'p' but the same architecture shown in the same plot? Each plot in Figure 3 shows the 84 different classifiers considered, with no repetition in the same plot. For each model, the optimal value p is the value of p that optimizes the MaxLogit-pNorm confidence estimator (with MSP fallback, as indicated in the legend). Then, each subfigure shows these models before and after post-hoc optimization, i.e., the first and third plots (baseline) actually show the models using the MSP baseline, while the second and fourth plots show these same models after optimization. We have revised the caption of the figure to clarify this point. Roughly speaking, the figure shows that, before optimization, the selective classification performance of many classifiers is bad and all over the place, while after optimization it improves significantly and becomes much better correlated with accuracy, which makes sense. Thus, “broken” confidence estimators have been fixed. >**Q8:** In Sec 4.3, it states: "This robustness can be explained by a strong correlation between the selective performance in the original test set and under distribution shift." Could you elaborate on what this sentence means? The sentence was indeed confusing; thanks for pointing it out. We have significantly rewritten Section 4.3 to make all our points clearer. The corresponding part now reads: “First, we evaluate the performance of MaxLogit-pNorm on ImageNet and ImageNetV2 for all classifiers considered. Figure 4a shows that the NAURC gains (over the MSP baseline) obtained for ImageNet translate to similar gains for ImageNetV2, showing that this post-hoc method is quite robust to distribution shift.” We hope we have satisfactorily addressed all of the reviewer’s concerns and would be happy to answer any other questions.

Response to Reviewer L98u (1)

Thank you very much for your constructive feedback. Please find our responses below. > As per my understanding, the paper proposes to optimize hyper-parameters of selective classification methods based on a hold-out dataset. This process is never formally defined in the paper. We kindly request the reviewer to take a look at the first paragraph of section 3.1 (Tunable Logit Transformations), where our hyperparameter optimization process is described: "The idea is to take any parameter-free logit-based confidence estimator, such as those described in section 2.2, and augment it with a logit transformation parameterized by one or a few hyperparameters, which are then tuned (e.g., via grid search) using a labeled hold-out dataset not used during training of the classifier (i.e., validation data). Moreover, this hyperparameter tuning is done using as objective function not a proxy loss but rather the exact same metric that one is interested in optimizing, for instance, AURC or AUROC." Specific details on the optimization of T and p used for our empirical results are given in Appendix D.1 (mentioned in the last paragraph of section 4.1). The size of the corresponding hold-out set used is also mentioned whenever a dataset is introduced (Section 4 and Appendix E). > Moreover, the paper does not consider potential (negative) side effects of this optimization procedure. Especially under distribution shift, deliberate calibration towards an in-distribution validation dataset might be more harmful than simply performing selective classification with a pre-defined threshold. We kindly request the reviewer to take a look at the first paragraph of section 4.3 (Performance under Distribution Shift), where this important question is addressed. Our goal is to investigate whether optimizing a post-hoc confidence estimator using the in-distribution data harms selective classification (SC) performance under distribution shift. In Table 3, we show using the SAC metric (maximum coverage allowed for a given target accuracy, in this case chosen as 80.86%, the original accuracy without distribution shift) that this optimization is not only **not** harmful under distribution shift, it is actually beneficial, since the coverage drops less than without post-hoc optimization. > Section 4.3 which talks about performance under distribution shift. It appears like the finding that better in-distribution classification performance leads to out-of-distribution performance is also not new and discussed in [1]. Although the paper does provide the exact numbers of how strongly performance degrades, the take-away message beyond the effect already introduced in [1] appears limited. Please note that the paper [1] provides empirical results for a specific performance metric, namely, full-coverage accuracy. It is generally an open question whether the same correlation holds for other performance metrics, such as calibration and SC metrics. Indeed, as mentioned by the reviewer in the previous point, it is **not true** that an intervention that improves in-distribution calibration will also tend to improve out-of-distribution calibration. But in section 4.3 we show that gains from post-hoc optimization in in-distribution SC performance **do** translate to gains in out-of-distribution SC performance, a result which does not follow from [1]. Additionally, considering all models after post-hoc optimization, we investigate in the revised section 4.3 (also in the new Appendix K) whether SC performance itself (as measured by NAURC) is robust to distribution shift. More precisely, we noticed that the SC performance of a classifier degrades under distribution shift; however, the (full-coverage) accuracy also degrades. It turns out that the increase in NAURC is entirely explained by the accuracy drop. This is not obvious: it is conceivable that distribution shift could produce an even higher degradation to NAURC (which would manifest itself by an RC curve that is higher at low coverage, even if we “scale” the curve to normalize for accuracy). Given the above points, we kindly request the reviewer to reconsider their opinion about the contributions of this section.

Response to Reviewer L98u (2)

> Neither temperature scaling nor logit normalization are new concepts but have been shown in past work to help reduce overconfidence and to improve calibration. Therefore, the paper does not provide novelty in terms of new selective classification approaches. In principle, the goal of the paper was not to propose new methods for selective classification, but rather to benchmark the existing methods across a large number of classifiers, instead of just a few classifiers as done in most papers. We considered a total of 20 methods by combining a logit-based confidence estimator with a tunable logit transformation, which were then applied to 84 ImageNet classifiers. Such an extensive study has never been done before. However, in the course of performing this benchmark study, we could not avoid making some contributions to selective classification methods. Namely: * We propose to apply logit normalization as a post-hoc method, while the original method was proposed to be applied during training; * We propose logit normalization with a general p-norm, while the original logit normalization only considered p=2. To the best of our knowledge, this extension to logit normalization had not been proposed before; * We propose to tune the confidence estimator (either the temperature or the p-norm parameter or both) using directly the AURC, which is also a novel idea. Note that conventional TS is optimized using the NLL; * Combining every logit-based confidence estimator with every applicable tunable transformation (among the ones we considered) produced several methods which had not been proposed before. Indeed, the best method found in our evaluation is a combination of MaxLogit and pNorm, which is not obvious and was entirely unexpected to us. Note that this differs significantly from applying logit normalization together with the softmax function as originally proposed in (Wei et al., 2022), which additionally requires temperature optimization (nor required by MaxLogit). Moreover, after picking the best post-hoc confidence estimator method for each classifier, our extensive evaluation enabled us to benchmark models. One of the conclusions—which to best of our knowledge is entirely novel in the selective classification literature—is that, across models, after post-hoc optimization, there is no tradeoff between selective and full-coverage accuracy, i.e., the best selective accuracy is obtained simply by the model with the best accuracy. Given the above points, we hope the reviewer agrees that the paper provides significant novelty to the selective classification literature. > It is unclear to me what the take-away message from Tables 1 and 2 are. Based on these results, it is unclear when we expect pNorm to work and whether architecture (and if so, which exact part of it) do prevent us from improving performance with pNorm. These 2 tables together (now unified as Table 1) show that: - an improvement is possible for some but not all classifiers; - when an improvement is possible, some confidence estimation methods work better than others. However, to really find out the best method (among the ones considered), we have to compute their average performance across all classifiers, which is done in Table 2, but we have to be careful to perform a fair comparison. Thus, Table 1 also serve to give an empirical motivation to the methodology described in section 3.2 (Evaluation Metrics). From these results (together with the data efficiency results in Appendix F), MaxLogit-pNorm emerges as a winner. However, these are empirical results. It is an open question to identify exactly which aspect of a model architecture or training regime renders it improvable (or non-improvable) by post-hoc optimization in general (or pNorm in particular). We believe this is an interesting question for future work that might inspire other researchers. Nevertheless, from the point of view of choosing a confidence estimation method for a particular classifier, it is a simple matter of experimenting to see whether it improves performance or not, similarly to optimizing a hyperparameter (as discussed in section 3.2.2), but with the benefit that it can be easily applied post-hoc, without retraining.

Response to Reviewer L98u (3)

> Section 4.2 suggests that the described post-hoc SC methods can fix "broken" confidence estimators. As per my understanding, NAURC was introduced to remove the E-AURC's accuracy sensitivity. If the proposed post-hoc fix works, would that not remove the the need for the NAURC score as accuracy now determines SC performance (i.e., accuracy and calibration are correlated)? The NAURC metric is useful whenever we wish to make comparisons of SC performance across different problems (where a problem includes the trained classifier, its confidence estimator and the test dataset). So it is a useful metric to evaluate confidence estimators while averaging performance across several classifiers, which is what we do in Section 4.1. It is also a useful metric to help us visualize the gains of a confidence estimator over the MSP baseline, which is what we do in Fig. 3(a) of Section 4.2. Now, if we are instead evaluating *classifiers*, each of them with the proposed post-hoc fix, then the reviewer is correct that in this case the AURC versus accuracy curve in Fig. 3(b) is probably more useful. Nevertheless, the NAURC could become useful again even in this case if we wish to compare classifier architectures while averaging SC performance across different datasets. (Since the reviewer mentioned calibration, we would like to remind that selective classification is not equivalent to calibration: as shown in previous work, there are cases where improving calibration harms selective classification and vice-versa.) We hope these responses address the reviewer's concerns and will be happy to answer any other questions.