In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defer

Summary

This paper considers the problem of learning to defer, and propose a softmax-based surrogate loss for the task, which is consistent and can be well-calibrated. The paper theoretically proves that one needs asymmetric losses for a bounded probability estimate---which in turn leads to a better calibration properties---(Theorem 1), and show the consistency of the proposed loss (Theorem 2). The paper also gives the risk transfer bound to the original zero-one loss (Theorem 3). Empirically, the proposed surrogate function enjoys the best of both worlds, achieving low error rates and good calibration properties.

Strengths

- Theoretical contributions of the paper is very good. Theorem 1 is a strong result, which may be useful in guiding the design of other softmax-based estimators for nonclassic (e.g., composite) learning tasks. Other two are somewhat standard, but are very essential in guaranteeing that the proposed loss works as desired. - The proposed asymmetric softmax-parameterized loss is very simply designed. The loss can be implemented very easily and may not introduce too much computational overhead to the training procedure. - The empirical performance of the proposed loss function is strong, consistently achieving the best performance over the tested setups. - The paper is very clearly written, especially for a paper that is heavily loaded with various notations.

Weaknesses

- It is not clear **why** we want to stick to the softmax-based losses from the first place. It is definitely cool that one can give a better calibrated softmax-based loss, but what is the reason that we need one when we have a decent non-softmax loss? We only know post hoc that such estimator has some empirical benefits, but the reason why it works better than non-softmax one is still at mystery. In the manuscript, authors try to justify the effort saying "Given the wide use and practical advantages of softmax parameterization (...)," but the reason is never spoken out clearly. For a better understanding of the advantages of the proposed loss, we need a more detailed explanation (or maybe a good reference). - I find that the empirical validation of the loss function to be somewhat limited. I don't usually ask for more experimental validations for theory papers. However, I think it is quite essential for this paper specifically, because none of the theoretical results actually explain why the proposed loss function should work better than the OvA loss. The main experiments are on the synthetic CIFAR100 dataset, with an additional figure on CIFAR10H (where are the "error rate" results for CIFAR10H?). This is very limited; [42] performed experiments on CIFAR10H, Hatespeech, Galaxy-Zoo, and HAM10000; [28] has experiments on CIFAR10H, CheXpert, and Hatespeech. In this sense, having only CIFAR100 results stated explicitly as numbers seems rather inadequate, as we cannot really check whether the baselines have been implemented/tuned correctly.

Questions

- What is the reason we should prefer softmax-based losses? - Please provide more experimental results on other datasets. - Please provide explicit numbers for the CIFAR10H experiments; this is for the sake of checking whether the baselines have been implemented and tuned correctly.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

2 fair

Limitations

The limitations are adequately stated in the manuscript, but only in the appendices.

Summary

In this paper, the authors investigate the task of probability forecasting in multi-class classification with an expert deferral option (L2D). They address the issue of unbounded and invalid estimates of experts' accuracy that often arises in probability estimation for L2D. Furthermore, the authors highlight that this problem persists when using a loss function with a symmetric structure, such as the commonly used softmax cross-entropy loss. To overcome the problem of unbounded probability estimates and support the use of softmax parameterization in L2D, the authors introduce a novel asymmetric softmax parameterization. They demonstrate the consistency of the induced loss function and provide theoretical evidence that this loss function will never encounter the issue of unbounded estimation. The authors validate their proposed method through experiments conducted on multiple benchmark datasets, utilizing various evaluation metrics. The experimental results provide justification for the effectiveness of the proposed approach.

Strengths

1. The authors successfully enable the utilization of softmax parameterization in L2D, which is popular but discouraged in previous works. They provide both theoretical and experimental justifications for this approach. The proposed loss formulation is intuitive and straightforward to implement, requiring only the addition of a single output without modifying the scoring outputs of the classifier. 2. The authors present a theoretical analysis highlighting the unbounded nature of symmetric losses in L2D. They demonstrate that the proposed loss function benefits from its asymmetric structure, emphasizing the non-trivial nature of their work. 3. The authors offer theoretical guarantees regarding the proposed loss function, ensuring that it will never produce an unbounded probability estimator. This rigorous analysis eliminates any concerns regarding unbounded behavior in the proposed method. 4. This work includes several figures that effectively compare the proposed method with related approaches. These figures illustrate the issues associated with unbounded probability estimators, providing persuasive evidence to aid readers in understanding the main points of the paper. 5. The authors thoroughly validate the performance of the proposed method using a variety of evaluation metrics and settings. Their results demonstrate that the proposed approach is comparable to, if not superior to, baseline methods not only in the context of probability forecasting but also in predictive tasks without coverage constraints.

Weaknesses

The authors are encouraged to provide a finite sample analysis, specifically an estimation error bound, to complement their theoretical analysis and ensure completeness. Considering that the newly proposed asymmetric softmax parameterization may alter the Lipschitz constant of the score function "g," the estimation error bound becomes a non-trivial aspect worth investigating. Additionally, combining this estimation error bound with the proposed regret transfer bound could potentially yield a more direct and intuitive result.

Questions

In Theorem 2, it is proved that the proposed loss will never have an unbounded probability estimator. Though the proof is clear, can you provide a more intuitive explanation of such a positive result?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

N/A

Summary

This paper studies the learning to defer setup where you have to defer to an expert if the classifier is likely to be wrong. It has been shown that softmax based consistent estimators for the learning to defer losses do not provide calibrated probability estimates for the likelihood of deferring. One other work proposed non-softmax based losses which provided calibrated probability estimates but it is not known if any other softmax based loss can lead to calibrated estimates. This work shows that softmax is not the issue of the lack of calibration but using symmetric loss functions is. This work then proposes a softmax based asymmetric loss function that leads to calibrated and bounded probability estimates. This work also experimentally verifies the effectiveness of using their loss function.

Strengths

- I find the idea of the paper really interesting. Since softmax based losses are the most commonly used loss functions, it makes sense to be able to use those loss functions for learning to defer framework. Moreover, this work identifies the fundamental issue present in the original loss function which was leading to poorly calibrated probability estimates. I find this insight on using asymmetric loss functions interesting.

Weaknesses

Right now, the paper sometimes becomes hard to follow as there are a lot of notations used. One thing that could be improved is having a common notation section in the beginning which could be used to look up symbols. Overall, the presentation of the paper could be improved.

Questions

In definition 1, R^c loss is not defined?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes

Summary

This paper shows that the miscalibration of the softmax-based surrogate loss for learning to defer is due to its symmetry. Instead, an asymmetric softmax-based surrogate loss is proposed and proved to be both calibrated and consistent. More generally, they reveal the connection between miscalibration and the symmetry of the used loss function and propose to design L2D surrogates based on asymmetric multi-class loss functions.

Strengths

The paper is well-organized and enjoyable to read though the notation is a bit heavy. They not only give a careful analysis of softmax-based loss functions but also provide very interesting insights regarding the miscalibration of surrogate loss functions for learning to defer and the symmetry of the multi-class loss functions they are based on.

Weaknesses

As stated in the paper, the 0-1 deferral loss studied here is not the most general version. For important settings including where the expert is a bigger model with higher accuracy but larger inference cost than the base model, an additional constant cost is required to reflect the extra inference cost. [31] shows that both [28] and [42] suffer from underfitting if the constant cost is non-zero. I am wondering if the proposed asymmetric softmax-based surrogate loss can be generalized to this setting and if it will have similar issues.

Questions

1. Could you give more details on how Definition 3 is derived based on the asymmetric softmax function? minors: 1. The notation of the risk is different in Definition 1 from in (1). 2. At the end of line 97, the transformation should map from R^{K + 1}. 3. It is better to specify the section when referring to the appendix.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

See Weaknesses.

Summary

The paper studies the learning to defer (L2D framework), where one can defer to an expert decision when unsure about the model’s prediction, and a cost is incurred when either the prediction is wrong or when one defers to the expert and the expert makes a mistake. The paper builds on top of prior work that shows that softmax parametrization can lead to unbounded probability estimators. While prior work provided a one-vs-all (OvA) formulation to solve this issue, this paper shows that the issue lies in the symmetric nature of the softmax function. The authors then proceed to introduce asymmetry in the softmax function and show that this leads to both bounded probability estimators and improved performance.

Strengths

1. The paper is well-written and the idea is somewhat novel. 2. I enjoyed reading the theoretical analysis in both the main paper and the appendix. 3. I like how the method do not have a lot of additional hyper-parameters to tune — this makes the method simple and do not feel over-engineered.

Weaknesses

1. Asymmetric softmax functions have studied before, although in a different context. For example, [1] suggests LDAM loss aimed for long-tailed or imbalance datasets. Comparison/discussion with this and possible other asymmetric variation of softmax would be appreciated. 2. The experiments involve only small-scale datasets like CIFAR-10 and CIFAR-100. While many methods perform well on these, they often do not scale well. Additional datasets would both (1) make the problem setting more appealing and (2) improve the standing of the paper. [1] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, Tengyu Ma. Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, NeurIPS 2019, https://arxiv.org/abs/1906.07413

Questions

1. How is the learning to defer paradigm comparable to selective classification [1]? In particular, one can integrate the reject option into the classifier. 2. In the learning to defer problem, a possible generalization is also associating a cost (possibly small) with deferring the prediction, even when the expert is correct, because using an expert instead of the model should be costly and avoided. Curious if any related work has considered this scenario. [1] Yonatan Geifman, Ran El-Yaniv. Selective Classification for Deep Neural Networks. NeurIPS 2017

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

N/A

Response

I thank the authors for addressing the concerns. I would like to keep my score.