When Does Confidence-Based Cascade Deferral Suffice?

Summary

The paper consists of 2 parts. Part 1 contains theoretical analysis of when confidence-based deferral rules for cascades of 2 or more models succeed or fail, based on a proposed risk function (equation (1) in section 3) presenting a tradeoff between accuracy and computational cost of invoking subsequent models. Part 2 proposes new deferral rules that are more sophisticated than existing confidence-based deferral rules using a machine-learned postdoc model. Experiments attempt to justify claims made in part 1 and to show that posthoc-based deferral rules produce better results than others.

Strengths

The paper provides a number of edge cases, both good and bad, together with good explanations on those edge cases. References to previous works are excellent. However, I cannot claim my expertise in this area.

Weaknesses

I have found a few places that appear rather puzzling in the followings: 1. Lemma 4.1 line 176: it is unclear what authors mean by "produce the same ordering over instances x". The lack of a precise formula makes it unnecessarily difficult to follow the proof for lemma 4.1 and to verify the proof, because I did not know what end result to expect. Eventually I understood what that term meant after carefully checking the appendix. But it was via a few definitions in the proof in the appendix, but not in the lemma statement itself. The statement definitely needs revising. 2. Section 4.1 seems to have a technical flaw in reasoning, despite that Lemma 4.1 appears to be correct. Consider functions $R1(c, c^{(1)}) = R(r_{conf} | c, c^{(1)})$ and $R2(c) = R(r^* | c)$. Lemma 4.1. shows a condition when the accuracy-deferral curve of $r^*$ matches with that of $r_{conf}$. But that is an analysis with respect to accuracy, not risk $R(\cdot)$ defined in (1). For a given $c$, for example $c$ represents a trade off between accuracy and inference speed, $R2(c)$ is constant but Lemma 4.1 does not guarantee that the lowest value of $R1(c, c^{(1)})$ over $c^{(1)}$ cannot go lower than $R2(c)$. To me, it only makes sense if we assume that the algorithm sees $c$ as a variable and we consider $c^*, R^* = \[arg\]min_c{R2(c)}$ instead. However, in such cases there are 2 issues: (1) Lemma 4.1 still does not guarantee that across all possible $(c, c^{(1)})$ pairs, the only time when $R1 = R^*$ is when $c=c^*$, and (2) if $c$ is a variable then how do you take practical considerations like inference speed into the analysis? While the subsequent success/failure cases in 4.2 seem intuitive to me, I just don't see how Lemma 4.1 can be used to convincingly explain them. 3. The general idea proposed in 4.2 is somewhat weak as well. In order to make a deferral rule for a cascade of 2 models to work better, you need to create a third model, i.e. the posthoc model g(x)? Wouldn't that incur an additional computational cost, meaning the risk function in (1) has to be redefined? I do not see any quantisation of that effect in the experiments. In addition, wouldn't introducing g(x) also mean that you have given a bit of training power to complement model 1 and 2, whereas the general setting of the paper is to treat model 1 and 2 as pre-trained blackboxes not to be tampered with (e.g. line 244-246)? 4. Section 5 presents all results in terms of accuracy-deferral curves. However, I cannot see any curve or any piece of information at all mentioning the risk $R(\cdot)$ in (1). This is puzzling. On one hand, sections 3 and 4 discuss minimising risk in (1) which can be seen as a tradeoff between accuracy and computational cost. On the other hand, all experiments only present information related to accuracy. I am not sure how those results can completely justify sections 3 and 4.

Questions

I need to be able to see that the paper has an objective and that every section is consistent towards that objective. At the moment, the three sections 3, 4 and 5 do not appear to be completely aligned with each other. If you could convince me where I am wrong in the Weaknesses section above, and show me some form of consistency throughout the paper, I will change my opinion. It is unclear to me whether the constant cost $c$ on line 117 in (1) is to be treated as constant or not. It appears to be constant at first (and it makes sense to do so from a practical point of view). But then in section 4.1, $c$ seems to be treated as a variable that can be optimised away. What is the point of having an accuracy-deferral curve for $r^*$ over $c$ if $c$ is constant?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

There is no societal negative impact from the paper.

Summary

This paper explores the cascade deferral problem, which is an issue in the context of machine learning models arranged in a cascading order, where a decision needs to be made about whether to defer the processing of data from one model to the next in the cascade. The challenge is to optimize the deferral decision to achieve the best trade-off between computational costs and model performance. Previously, this problem was often addressed using confidence-based deferral rules. This approach involves deferring data to the next model in the cascade when the first model's confidence in its prediction falls below a certain threshold. However, the paper identifies limitations with the confidence-based deferral method. Specifically, it may underperform in certain settings, such as when the second model in the cascade is a specialist model, or when there's label noise in the training data. To overcome these limitations, the authors introduce post-hoc deferral rules. Unlike the confidence-based approach, post-hoc deferral rules use additional information, beyond just the confidence of the first model, to make deferral decisions. These rules are trained and optimized to provide better accuracy-cost trade-offs. The authors compare the performance of confidence-based and post-hoc deferral rules under various experimental settings. They use datasets like ImageNet and CIFAR 100, with settings including a specialist model scenario, label noise, and distribution shift. They find that post-hoc deferral significantly outperforms confidence-based deferral in scenarios involving a specialist second model and label noise. However, they also identify potential overfitting issues with post-hoc deferral, highlighting the need for careful capacity control.

Strengths

**Originality**: The paper introduced a post-hoc deferral scheme that utilizes a Bayes-optimal deferral rule. In particular, it addresses issues not resolved by the traditional confidence-based deferral methods. **Quality**: The authors effectively utilize mathematical proofs and models to construct and explain their deferral schemes, and they validate these models through extensive experiments on established datasets such as ImageNet and CIFAR 100. **Clarity**: The paper is well-structured and clearly written. The paper's theoretical concepts are explained with clarity and are substantiated with illustrative figures, while the experimental design and results are presented in detail. **Significance**: The research addressed a critical limitation in the commonly used confidence-based deferral schemes. Confidence-based methods, while widely used, tend to underperform in certain situations such as in specialist settings or when there is label noise. This research introduces and investigates the efficacy of post-hoc deferral models as a solution to this problem, offering a more optimal and efficient method of deferral. The paper's analysis of the limitations of the confidence-based methods and the demonstration of how the post-hoc deferral models overcome these issues are essential for further research in this area.

Weaknesses

**Limited Dataset and Task Diversity**: The empirical results of this paper are primarily based on two datasets: CIFAR-100 and ImageNet. Additionally, the experiments were conducted on image classification tasks only. Expanding the scope of datasets and including other tasks, such as object detection, segmentation, or even venturing into different domains like natural language processing or audio processing, could have provided a more comprehensive evaluation of the post-hoc deferral models. **Generalizability**: The authors point out that post-hoc models can overfit and fail to generalize well, even when controlling the capacity of the model.

Questions

The authors acknowledge that post-hoc models can overfit and struggle to generalize. Could the authors provide some intuitive explanation for the reasoning behind this issue?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have adequately addressed the limitations

Summary

This paper systematically investigates why and when the confidence-based deferral rule work, and particularly, identifies cases when it fails. To enable this investigation, they provide a theoretical characterization of the problem and the optimal deferral rule. They also provide a post-hoc solution that can work well in cases when the confidence-based method fails.

Strengths

The paper is clearly written. The paper provides a nice theoretical framework for studying the deferral rule in general. The paper provides lots of useful insights, supported by both theoretical and empirical evidence.

Weaknesses

The listed conditions about when the confidence-based method fails are not that surprising, this leads one wondering the significance of such a contribution if the findings are already something intuitive that need not proof. Anyhow, this might not count as a weakness, as it is also reassuring that the theory does produce intuitively sensible results.

Questions

This deferral rule is about selecting models, while there is also this line of research about selecting samples for the noisy data case. Basically, for the data with noisy labels, one would want to select only the clean ones to do the training, and there are also many confidence-based methods to do this selection. I am wondering how these two lines of research connect with each other, and can you similarly identify cases when the confidence-based method fails in label denoising? It would be good if the authors can provide some discussion/ideas about how to identify whether it is good to use the confidence-based method in practice. Basically, is there a unified way to identify the failed cases from the data, instead of checking if the data have the issues related to each failed case?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors have addressed the limitations.

Summary

The authors present a theoretical analysis for the Bayes optimal deferral rule for a cascade of K=2 classifiers and a certain population risk. Based on this rule, they characterize when confidence deferral rule using exact posterior probabilities is similar to the Bayes optimal deferral rule is some sense. Based on their analysis, the authors (informally) discuss cases where (practical) confidence deferral rule may or may not be sufficient, and proposed several post-hoc methods for the latter. The insights and the methods are examined empirically.

Strengths

I find the paper very interesting and the analysis to be solid. The theory is important and the insights and practical guidance that it provides look useful.

Weaknesses

The empirical coverage may be somewhat improved, e.g., by using practical datasets without the controlled modifications (such as the dog-specific classifier). I also wonder whether there is some benchmark that can be used for comparing the new post-hoc methods to existing ones.

Questions

Please improve the explanation for Lemma 4.1. State what you mean in "the same ordering" in the main body of the paper and not only in the proof in the appendix. The text below Lemma 4.1, and actually in several other places in the paper, needs to be more accurate when you claim optimality of confidence-based deferral rule based on your analysis --- as by saying so, you assume that the classifier's softmax values are exactly the true posterior probabilities, which is obviously not the case in practice [21]. In Figure 2, first row, how do you explain the fact that plain confidence deferral oftentimes outperforms the two post-hoc methods Diff-01 and Diff-prob? Furthermore, Diff-prob in fact is consistently worse than the other methods in all 3 rows in Figure 2, can you explain why? A minor comment: In Algorithm 1, when you write predict inside the loop, letting it be followed by a "break" command would make the scheme clearer.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Summary

In this paper, the authors study the problem of confidence-based cascade deferral for the classification task. They first proposed the formulation of Bayes optimal deferral rule, which relies on both the base model and its successive model, and then proposed several baselines that works by mimicking the Bayes optimal rule. They further analyzed confidence-based methods and proposed the condition for their consistency. Given the fact that these baselines' computational costs are not acceptable, they turn to propose a post-hoc method guided by the baselines and analyzed its consistency under different conditions. Experimental results validated the performance of the proposed method.

Strengths

1. The authors studied the optimality condition for cascade deferral rule and show the formulation of the Bayes optimal rule that relies on the confidence score of two models, which takes an intuitive form that implies the limitation of confidence-based deferral. 2. The authors gave several baselines to approximate the Bayes optimal rule, and further proposed the post-hoc versions for them to resolve the computational issues. 3. Theoretical analyses are conducted for both the confidence-based method and post-hoc deferral rule under various conditions.

Weaknesses

1. Though the connection between this work and learning to defer/ classification with rejection is revealed, it is not thoroughly investigated in my opinion. In fact, learning to defer and classification with rejection are shown to be equivalent to the ordinary multi-class classification problem [1, 2]. In the post-hoc regime, I think it is worth trying to reduce the training of the cascade deferral rule to a multi-class classification problem by setting the maximum confidence of each model as a pseudo posterior probability. 2. As stated in the limitation section, the analyses are conducted on a case-by-case basis, indicating that there is still progress needed to develop a consistent and computationally friendly deferral rule. While the experimental results suggest that the proposed method is comparable, it is still a matter worth addressing. Conducting a finite-sample analysis could significantly alleviate this concern. [1]. Mohammad-Amin Charusaie, Hussein Mozannar, David A. Sontag, Samira Samadi. Sample Efficient Learning of Predictors that Complement Humans. ICML 2022. [2]. Yuzhou Cao, Tianchi Cai, Lei Feng, Lihong Gu, Jinjie Gu, Bo An, Gang Niu, Masashi Sugiyama. Generalizing Consistent Multi-Class Classification with Rejection to be Compatible with Arbitrary Losses. NeurIPS 2022.

Questions

Please see the Weaknesses section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The limitations are adequately analyzed in Appendix G.

Thank you. Points 1, 2 and 4 cleared. Point 3 remains for discussion.

Thank you for the clarifications. >> Section 4.1 seems to have a technical flaw in reasoning, despite that Lemma 4.1 appears to be correct… > There appears to be a misunderstanding which we’d like to clarify... Right. The Lagrangian manipulation to connect between the unconstrained risk in (1) and accuracy with a constraint. Clearly, without this fact it is difficult to link section 3 with sections 4 and 5. This is a crucial missing bit, which clears both my points 2 and 4. Why did it not appear in the original version? It would have been a lot easier to read had the risk-accuracy equivalence got established in section 3. Since points 2 and 4 are major, I am ready to increase my ratings by the way. >> cost of the posthoc model g(x) > Our post-hoc model is a simple feedforward network and is much smaller than the two base models... I understand your argument. But your paper's title is `When Does Confidence-Based Cascade Deferral Suffice?`. To a normal reader, the title suggests that the cascades of interest contain pretrained models plus some thresholding on the model confidences, no additional training is needed. A posthoc model would require the user to train additionally. Under a research context this may be fine. But in real world, this additional requirement can be costly due to resources and teams involved. It may be better to improve the pretrained models instead. I am not saying the idea of using a post-hoc model is wrong. I am saying that in my practical view point, it somewhat backfires. In addition, a hidden assumption here is that the posthoc model g(x) must be very lightweight compared to the pretrained models. But in very high speed applications where the pretrained models can be marginally slower than the posthoc model g(x) then this assumption will break.