Summary
The paper presents 5 requirements for multi-threshold metric for Selective Classification and a novel metric to evaluate selective classifiers called AUGRC. The proposed metric satisfies 5 requirements that are not met by current approaches. The proposed metric changes the rankings on 5 out of 6 datasets considered by the authors.
Strengths
The paper's main strengths are:
1. the paper aims to tackle a long-standing concern in abstaining classifiers literature, i.e. how to evaluate with a single measure these classifiers
2. the theoretical derivations seem sound
3. the contribution is well framed within current literature
Weaknesses
The main concerns of the paper are:
1. The empirical evaluation can be improved
2. The interpretability of the proposed measure is not that straightforward
3. The paper presentation can be improved.
Regarding the empirical evaluation, I have a few remarks.
* there are some contradictory lines, which should be double-checked: for instance, in lines 268-269, it is not clear to me why you claim that AURC erroneously favors DG-RES over MCD-PE, as DG-RES's performance seems to be better than MCD-PE (accuracy-wise and ranking-wise). Similarly, it is not clear to me why in Figure 4 the authors state DG-RES is favored "despite a lower classification performance and ranking quality", while in Figure 4.a is reported a higher accuracy and higher $AUROC_f$ for DG-RES;
* second, the authors never specify whether they correct for multiple outcomes testing (e.g. using Bonferroni correction). I think since the authors are performing multiple pairwise ranking tests, they also have to account for this.
Regarding the interpretability requirement, the authors claim (correctly) that the AUROC can be interpreted as "the probability of a positive sample having a higher score than a negative one". I did not fully grasp what is the straigthforward interpretation of the AUGRC score in this context (lines 206-208).
Regarding paper presentation, the paper heavily relies on acronyms. I would personally reconsider this choice to improve the overall readability.
Questions
I have a few questions for the authors:
* [Q1] can the authors clarify my concerns regarding the empirical evaluation?
* [Q2] can the authors discuss further my concern regarding interpretability? Is there a straightforward interpretation of the AUGRC score?
* [Q3] when considering a degenerate abstaining classifier that always abstains, i.e., whose confidence score is always zero, is the AUGRC 0 (as the one obtained by a perfect classifier)? If so, is AUGRC favoring classifiers that tend to be underconfident and asbtain too much? Is there a way to avoid such a behaviour?
* [Q4] if I correctly understand how AUGRC works, why do not the authors consider the _classification quality_ by [Condessa et al, 2017] instead of the Generalized Risk Score? Such a metric also considers the failures that are correctly rejected and could avoid favoring abstaining classifiers that over reject.
Typos:
line 128: schore $\to$ score
Limitations
The authors do not directly discuss limitations of their proposed approach in the main paper. For instance, I think a brief discussion regarding the computational time required to compute AURGC should be included in the main paper.