Further clarification
> In terms of the metric, I do not understand exactly why the metric by Condessa et al is not applicable in "evaluating" abstain classifier? To me it seems like the most natural and basic way to evaluate a selective classifier. Could the authors please let me know what the proposed score does, that current methods cant exactly.
We did not say anywhere that Condessa et al.’s metric is not applicable in evaluating an abstaining classifier. We elaborate on when the counterfactual score is preferred over Condessa et al.’s score in lines 89–103 in our paper, including the following line:
_While [Condessa et al.’s] view is relevant to the training of abstention rules, it is at odds with black-box settings where the underlying predictions may still be executed even when the method abstains, motivating the counterfactual score._
The questions of (1) whether a score is natural and (2) what “current methods can’t do” are separate.
1. Our whole intro is devoted to justifying that the counterfactual score, as we defined it in our paper, can be useful (rather than Condessa et al.’s). Both metrics can be viable metrics for evaluating abstaining classifiers, but they simply evaluate the classifiers in a different manner. The two scores can be useful in different scenarios, and we focused on scenarios in which our proposed metric can be useful.
2. Condessa et al. (or any other work, to our knowledge) do not discuss the black-box setting that requires a _statistical estimation_ approach, as they already assume full knowledge of whether the classifier would have classified each input correctly or not. But this is not known in any of the black-box settings we describe. Both Condessa et al.’s score and the counterfactual score require how accurate the classifier _would have been_ on its abstentions, and we do not know this in the black-box scenario like Example 1.1 (we have no clue about whether an API would get a 100% or 0% accuracy on inputs that it chose to abstain from making predictions). This unknown quantity is really the “counterfactual,” and thus we devote Sections 3 and 4 of our paper to discuss how we can _estimate_ the counterfactual score in the black-box setting. No other "current method" can estimate the counterfactual score (because it was never defined formally), and we further mention in Appendix A.3 that our method can further estimate Condessa et al.’s score in the black-box setting.
> I would like to hear an exact and precise example please.
To give a simplified example, suppose that we compare two abstaining classifiers on 100 data points, whose inputs are sampled uniformly on $[-1, 1] \times [-1, 1]$. Suppose that classifier A achieves a 1.0 accuracy on the left half of the input space ($x_1 < 0$) but a 0.8 accuracy on the right half ($x_1 \geq 0$). It abstains at an 80% rate on the right half, while it does not abstain at all on the left half. On the left half, the classifier makes 50/50 correct predictions on the left half; on the right half, it makes 8/10 correct predictions and 40 abstentions (for which it would have been correct 80% of the time).
Recall the definitions from Appendix A.3. The counterfactual score of classifier A is 58/60 * 0.6 + 32/40 * 0.4 = 0.9, whereas Condessa et al.’s score for this classifier is 58/60 * 0.6 + (1 - 32/40) * 0.4 = 0.66. Note that the classifier’s selective score is 58/60=0.97 while its coverage is 60/100=0.6.
Next, suppose that classifier B is the same as classifier A, except that it achieves a 0.6 accuracy on the right half. Then, classifier B’s counterfactual score would be 0.8, lower than classifier A, whereas Condessa et al.’s score for B would be 0.82.
If the evaluator needs to access the classifier’s hidden predictions on the right half, they would prefer classifier A, which has a higher accuracy had no abstentions were made. In Example 1.1, the right half would correspond to the inputs that the free-trial API chose to abstain on.
> Also I dont understand this sentence. So what is the point then? I am really confused now, could you please clarify this part. "These experiments are NOT meant to provide further justifications on why the counterfactual score is a viable alternative as a metric to existing ones; they are simply meant to show that our method for estimating the counterfactual score is sensible and works well in practice".
Because we developed an estimation approach, we need experiments to validate whether the estimator actually estimates the target quantity well on simulated (and real) data. We note that it is _not_ trivial whether this score can be estimated well in practice, given that it is an unknown counterfactual quantity. The real data experiments illustrate different comparison scenarios and what the estimated counterfactual score would be.
We hope that these responses clarify your concerns.