Rebuttal Part 2 for Reviewer vmdY
*"Q1. I think the authors are not discussing an implicit (and, in my opinion, quite relevant) assumption of their strategy, i.e. the errors at different levels of the hierarchy are assumed to be the same. However, I argue this is not exactly the case in real life. For example, failing to distinguish a golden retriever from a labrador differs from failing to distinguish a dog from a spider. Can the authors elaborate on this point?"*
That's a great question! In fact, we conducted all of our experiments using a hierarchical risk that considers scenarios like the ones you've illustrated. We chose not to include these results in the paper due to space constraints and for the sake of simplicity and readability.
We developed a hierarchical risk that considers mistake severity with respect to the hierarchy, as follows:
$R_h = 1 - \frac{\phi(LCA(\hat{v}, v))}{\phi(\hat{v})}$
Where $v$ is the ground truth node, $\hat{v}$ is the predicted node, $\phi(x)$ is the coverage of node $x$, and $LCA(x,y)$ is the lowest common ancestor of nodes $x$ and $y$. The most severe mistake occurs when $LCA(v,\hat{v})$ is the root of the hierarchy, resulting in a risk of 1. The higher the coverage of $LCA(\hat{v},v)$, the more $\hat{v}$ has in common with $v$ in hierarchical terms.
One can safely assume that in any plausible hierarchy the LCA of (labrador, golden retriever), for example, dog, has significantly higher coverage compared to the LCA of (dog, spider), resulting in the former misclassification having lower risk compared to the latter.
We provide below some of the results of our experiments on ImageNet, complementing Table 1 in the paper (all other tables and figures can also be made with this type of loss):
| | ImageNet-1k (1115 models) | |
|:--------------:|:-------------------------:|:-------------:|
| Inference Rule | hAURC | Hier. Gain |
| | $\times$ 1000 | (\%) |
| Selective | 24.98$\pm$0.29 | - |
| MC | 24.50$\pm$0.34 | 3.31$\pm$0.30 |
| Climbing | 24.04$\pm$0.31 | 4.71$\pm$0.22 |
The results align with those presented in the paper, where the risk is 0/1 loss. Climbing remains the inference rule with the best (lowest) hAURC and the highest hierarchical gain. It's theoretically possible to create examples where the hierarchical risk diverges from the 0/1 loss, such that models making less severe mistakes are penalized less by the hierarchical risk. However, in practice, the differences are smaller than we anticipated and are very highly correlated (which is interesting by itself). Do you think we should include these results and others with this risk in the revision?
[1] Ido Galil, Mohammed Dabbah, Ran El-Yaniv. What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers, ICLR 2023. https://arxiv.org/pdf/2302.11874
[2] Guo et al. On Calibration of Modern Neural Networks.
[3] Vladimir Vovk. Conditional validity of inductive conformal predictors.
[4] Cattelan and Silva. How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networks.