We thank the reviewer for the helpful comments. We agree with the reviewer and appreciate the conformal prediction community for their efforts and results, and we find the papers mentioned very inspiring. In the past few days, we implemented numerical experiments to compare our algorithm with the algorithm in Paper 2 on the 6 datasets that are of the highest dimensions among the datasets in Paper 2. Such conformal prediction benchmarks and detailed experiments will be included in our further versions.
1. We apologize for missing some of the conformal prediction literature, but indeed we tried our best for the literature review. We are willing to add more discussions as mentioned in our previous responses in the future. Among the conformal prediction literature, we aim to address the minimum assumption to keep both the individual calibration (or coverage) and the finite sample guarantee, reconciling the impossibility results for quantile calibration (see our Appendix B) as well as the impossible triangle in the field of conformal prediction. Theoretically, we don’t find any paper in the literature on conformal prediction that gives a better result with comparable or weaker assumptions (including the papers mentioned by the reviewer). We refer to the previous Author Rebuttal for details.
2. As for the (practical) performance metrics, although the quantile calibration task is very closely related to the conformal prediction task, the goals are still different. Conformal prediction aims to provide as sharp as possible prediction sets that cover the true outcomes with desired rates. While a precise quantile prediction guarantees the coverage rate, such a covering band may not be the sharpest. As is discussed in Appendix A, the additional sharpness regularization term could harm the goal of precise quantile prediction. For example, people may select 0.05 and 0.95 quantiles to construct a 0.9 coverage band, but the sharpest covering band probably would deviate from such quantiles. But a high-quality quantile prediction alone is of independent interest for many downstream tasks, such as the newsvendor problem. This points to a difference between the objectives of calibration and conformal prediction. As a result, the popular performance metrics applied in conformal prediction such as the average interval length and the coverage rate are not direct measurements for the quantile calibration problem. Some other measurements, for example, the independence between the coverage indicator and the band length in Paper 2, are only a necessary condition of the quantile calibration. We henceforth consider only the explicit measurements rather than such implicit ones in our paper.
3. Numerically, to further illustrate the issue of high-dimensionality, we selected the 6 highest-dimensional datasets that appear in Paper 2 and applied our algorithm and Paper 2’s Pearson-correlation-regularized algorithm. First, we note that the first step regression of our algorithm doesn’t have to be the mean regression but can be any regression algorithm. We apply a quantile regression algorithm as the first step, and then recalibrate the initial result using our nonparametric estimator. As for the metric, we select (a 90% confidence interval variant of) the Adversarial Group Calibration Error (AGCE, defined in Appendix D.1) to show calibration error in the worst calibrated part of the data. We list the results as follows (Paper 2’s OQR the former, and the latter ours):
meps_19:
0.071,
0.03
meps_20:
0.059,
0.031
meps_21:
0.043,
0.036
facebook_1:
0.041,
0.022
facebook_2:
0.011,
0.018
blog_data:
0.03,
0.024.
The numerical performance shows the advantage of our algorithm. Of course, AGCE is a conditional/individual performance measure; for marginal calibration objectives or other datasets, conformal prediction methods may have an advantage. Just like the general ML problem, we don’t expect a single ML model that performs universally well, but we do believe our algorithm provides a simple and efficient complement to the existing methods.
4. As for the high-dimensional cases, we here make some further explanations. On one hand, the belief that high-dim data can be represented by lower-dim features is common in modern machine learning. Both computer scientists and statisticians are making efforts to extract low-dim features from the original high-dim datasets. On the other hand, the tremendous impact of reducing the Lipschitz coefficient $L$ partly explains why our algorithm works even in the presence of high dimensionality (and also why there is a splitting procedure in the split conformal prediction). As mentioned by the reviewer, there are conformal prediction algorithms that don’t follow the split protocol, but to our knowledge, we provide a first explanation/justification for those split conformal prediction algorithms.
We thank the reviewer again for their helpful comments, which we believe consolidate the overall positioning of our paper.