Realizable $H$-Consistent and Bayes-Consistent Loss Functions for Learning to Defer

Summary

This work studies learning to defer, focusing on the single-stage and single-expert setting. It introduces a family of surrogate losses based on comp-sum losses [Mao et al., 2023b] and establishes their realizable H-consistency (under mild conditions). In addition, when the base loss is the logistic loss $\Psi_{log}$ or the generalized cross entropy loss $\Psi_{gce}$, H-consistency bounds are proven (under mild conditions) when the cost function of deferring is classification error. When the base loss is the mean absolute error loss $\Psi_{mae}$, H-consistency bounds are proven (under mild conditions) when the cost function of deferring is general. Note that H-consistency implies Bayes consistency. The results also close an open question raised by Mozannar et al., 2023. The relationship between realizable H-consistency and H-consistency bounds is then analyzed for learning to defer. Finally, the proposed surrogates are evaluated empirically and compared with existing baselines.

Strengths

**Originality** - A new family of surrogate losses based on comp-sum losses (focusing on $\Psi_{log}$, $\Psi_{gce}$, and $\Psi_{mae}$) [Mao et al., 2023b] for learning to defer (the single-stage and single-expert setting). - Conditions for realizable H-consistency are identified (Theorem 4.1). In particular, $\Psi_{log}$, $\Psi_{gce}$, and $\Psi_{mae}$ all satisfy the conditions. This result is more general than Mozannar et al. [2023] - When the cost function of deferring is classification error, H-consistency bounds are proven for base losses $\Psi_{log}$ and $\Psi_{gce}$ (Theorem 4.2). - When the cost function of deferring is general, an H-consistency bound is proven for base loss $\Psi_{mae}$ (Theorem 4.3). The results also close an open question raised by Mozannar et al., 2023 (Section 4.4). The relationship between realizable H-consistency and H-consistency bounds is then analyzed for learning to defer (Section 5). Related work is adequately cited and compared. **Quality** The submission is technically sound. Claims are well supported by proofs. **Clarity** The submission is generally clearly written and well organized. **Significance** The work closes an open question raised in previous work. Its work to connect both realizable H-consistency and H-consistency bounds might be useful in other learning settings.

Weaknesses

**Originality** - The realizable H-consistency result (Theorem 4.1) only applies to a subset of comp-sum losses. - The H-consistency results (Theorems 4.2 and 4.3) require case-by-case analysis. Is it possible to prove such results for all comp-sum losses? **Quality** - Some experiments in realizable settings can help confirm the realizable H-consistency result. **Clarity** - Section 5 is a bit unclear. Lines 300-302: Do you mean that in the realizable setting, all surrogate minimizability gaps vanish? Or is it only for comp-sum losses? **Significance** - The applicability of this work might be limited to learning to defer.

Questions

Besides my concerns above, here are some other questions: 1. Can constrained losses [1] be used as the base loss? 2. Lines 273-274: Can negative results be proven formally? [1] Y. Lee, Y. Lin, and G. Wahba. Multicategory support vector machines: Theory and application to the classification of microarray data and satellite radiance data. Journal of the American Statistical Association, 99(465):67–81, 2004.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations.

Summary

This paper considers the problem of learning to defer (L2D), where a classifier is allowed to defer a decision to an expert (possibly expensive to query) and trained to accurately predict while minimising the expert cost. A major contribution of this paper is establishing consistency guarantees for surrogate losses. Specifically, the authors derive a loss that encompasses existing surrogate losses. The authors provide sufficient conditions for the proposed surrogate loss to have realisable H consistency and for H consistency bounds. From the derived H consistency bounds, the authors show that certain choices of the expert cost leads to a surrogate loss that is both Bayes and realisable H consistent.

Strengths

The paper is very well written and does not have major issues in clarity. This paper seems to provide a general framework for analysing surrogate losses for L2D, and the established results appear to be novel. I am not actively working on L2D and my evaluation might not be accurate.

Weaknesses

Due to my lack of experience in the field, I find it challenging to adequately adjudicate the significance of the work. Realisable H consistency effectively assumes that there is a perfect classifier that does not need an expert. This preposition seems to be strong and diminishes the point of having an expert.

Questions

1. Why is H consistency important in L2D?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

The authors provide a framework of surrogate loss functions for learning to defer under the multi-class classification problem. By examining the deferral loss function and choosing different surrogates for the indicator functions, the authors provide a novel class of surrogate loss functions for learning to defer and prove the realizable $\mathcal{H}$-consistency and $\mathcal{H}$-consistency for some cases. The authors also verify their proposed loss with numerical results.

Strengths

1. The analysis is novel and provides new insights for learning to defer literature; 2. The paper addresses the problem of achieving $\mathcal{H}$-consistency and realizable $\mathcal{H}$-consistency simultaneously.

Weaknesses

1. (Major) The proposed loss is not practical in many critical settings when the cost to consult an expert is high; Both terms of the derived loss function contain the exact value of $c(x, y)$ while knowing $c(x, y)$ itself needs consulting the expert if $c(x, y)$ is related to the expert's response. Thus, training a model using the proposed surrogate loss requires querying not only the true label but also the expert's response to every single sample, which is somewhat against the motivation of deriving the problem of learning to defer. See the derivation of equation (2). (Addressed, improve my rating to 6) 2. (Minor) Some of the results are restricted to the case $c(x,y) = 1_{g(x) \neq y}$, which is limited compared to the vast problem settings of learning to defer; 3. (Minor) The presentation is a little confusing: there are three consistencies (Bayes consistency, $\mathcal{H}$- consistency, and realizable $\mathcal{H}$-consistency), while their relationships should be formally summarized into some propositions; the existing surrogate loss functions' consistencies should also be summarized into one table with indicator "yes", "no", or "not proved" for each consistency. 4. (Moderate) The numerical experiments don't specify the expert algorithm; the experiments are also very restricted to the case of $c(x,y) = 1_{g(x) \neq y}$.

Questions

1. The major concern of mine is that the training under the proposed loss function requires the full responses of the expert to the entire training dataset, which is undesirable for the L2D problem. Previous surrogate loss functions (nominated in Section 3.3) only require $c(x, y)$ if the model chooses to defer. The learning to defer problem is proposed to ease the pain of consulting experts, while I think the loss function proposed works against that principle. Is it possible to mitigate the training cost of the model? 2. All the remaining questions are addressed in the Weakness part and do not matter unless the first question is addressed.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

NA

Summary

This paper proposes considers the setting of learning to defer: a machine learning system can choose to either classify an instance or defer the decision to an expert which incurs a variable cost. The objective is to minimize the deferral loss of the system. To solve this problem, prior work has proposed surrogate losses with certain theoretical guarantees with respect to the original deferral loss. This work proposes the first surrogate loss RL2D that is realizable H-consistent, satisfies an H-consistency bound and thus is bayes consistent. This resolves an open problem proposed from prior work. Empirically, the authors showcase that the proposed surrogate exceeds or matches prior surrogates on three different datasets.

Strengths

Originality: The surrogate loss proposed in the work is novel as well as the proof technique for the theoretical properties. The derivation of the surrogate loss is different from prior work, however, it is not clear how the technique can be generalized to other settings. Quality: I have verified a good portion of the theoretical derivations and they seem sound. The experimental setup follows similar protocol to prior work and is sound. The paper is very strong theoretically. Clarity: very well written, clearly stating contributions of prior work and setting the stage for readers unfamiliar with the setting and the literature. Derivation and theoretical properties very well stated and easy to follow along. Significance: The paper settles an open problem from prior work at AISTATS and in turns I believe concludes (barring any breakthroughs) a line of work on deriving surrogate losses for learning to defer. I think this is important because now the community can focus on other settings and other considerations beyond theoretical consistency properties. However, I don't think the paper has a lot to offer in terms of techniques/methods for the community beyond the learning to defer problem as the derivation rely on some algebra of the deferral loss. Therefore, I don't expect this to be widely read by the community, but will be instead read in great detail by the community working on learning to defer and related problems.

Weaknesses

There are no weaknesses with regard to the theory in this paper beyond the generalizability of the approach taken to related problem settings. However, the experimental setting is quite limited in terms of showcasing the behavior empirically of the newly proposed method. For the use of the surrogate in practice, it is not clear in which scenarios (if any) is the surrogate superior to prior work. Moreover, it is not clear how do the theoretical properties help in practice.

Questions

- In which settings is the new proposed surrogate better than prior work? - How do the theoretical properties manifest in practice? Follow-up after author response: - the authors have done a good job answering the concerns, I believe this paper merits acceptance but I will maintain my original score as I am unsure about the level of impact of the paper.

Rating

6

Confidence

5

Soundness

4

Presentation

4

Contribution

2

Limitations

Yes they have.

Increase my rating to 7

Thank you for the detailed responses to my questions and concerns, as well as those in other reviews. I am satisfied with the reply and have increased my rating to 7.