Thank you for your reply
Thank you for your insightful comment.
**Add Q1/A1:**
We indeed have some applications in mind where the instance-specific rejection cost is required. For example, in safety-critical applications such as autonomous driving, we need to predict the suitable deflection angle for the tires to pass through an intersection based on sensor information. Obviously, the difficulty and risk faced by an intersection that can only be passed with a deflection angle from $5^{\circ}-10^{\circ}$ is not the same as an intersection that can be passed with a deflection angle from $10^{\circ}-40^{\circ}$, therefore they do not have the same rejection cost (rejection cost $2.5^{\circ}$ in the first intersection and rejection cost $15^{\circ}$ in the second intersection).
We will not take Theorem 2 as one of the main contributions of our paper. In this case, our main contributions in this paper are summarized as follows:
- We propose a surrogate loss function considering rejection as a binary classification process and give a condition of regressor-consistent that the classification calibrated binary classification loss is always greater than 0. In that condition, the optimal regressor can be derived by our method.
- We propose a definition of rejector-calibration and show that our method is rejector-calibration when the regressor-consistent condition is satisfied. Based on this, we further propose a weaker version of the condition allowing the classification calibrated binary classification loss to be greater than or equal to 0. In the weakened condition, the regression consistency can only be satisfied in the accepted instances, and regressor-consistent is still satisfied.
- We derive the theoretical analysis of the regret transfer and estimation error bounds for our proposed method, and extensive experiments demonstrate the effectiveness of our method.
**Add Q2/A2:**
We are really sorry that we have mistakenly linked [19] to the paper of [18], in our previous response. The correct links (in accordance with our originally submitted main paper) are as follows:
[18] Y. Geifman and R. El-Yaniv. Selective classification for deep neural networks. In NeurIPS, 2017.
[19] Y. Geifman and R. El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In ICML, 2019.
This mistake is because we removed duplicate references in our revised version (not uploaded yet) and thus the serial numbers of the original references were changed.
We really appreciate that you provided a wonderful visualized way (risk-coverage curve) to compare different methods for regression with rejection. Unfortunately, we cannot add images to OpenReview in the current stage, but we promise that we will definitely include these visualized figures in our final version.
**Add Q3/A3 and Q6/A6:**
We agree that the estimation of the uncertainty score is quite important to regression with rejection, which can be used as the metric to determine whether an example will be rejected. We also agree that the uncertainty score (the conditional variance) can be used to solve the problem we studied in this paper and such a method can be considered as a baseline. One intuitive baseline is to use the Plug technique proposed in [41], which we have already compared with (i.e., **MLP-Plug**). We will be much appreciated if you have other suitable methods to estimate the conditional variance and we will definitely try our best to include them for comparisons.
In fact, we believe that our proposed method is better than those baselines relying on the uncertainty score. By Occam's razor principle, calculating the uncertainty score is an intermediate, which is normally not the optimal solution, especially when the uncertainty score cannot be accurately estimated. In contrast, our proposed method gives a direct solution by taking rejection as binary classification. Therefore, our method is expected to achieve better performance.
[41] A. Zaoui, C. Denis, and M. Hebiri. Regression with reject option and application to knn. In NeurIPS, 2020.