Self-Calibrating Conformal Prediction

Summary

The authors propose a new scheme to obtain prediction intervals that are valid conditioning on the model output. The approach leverages the notion of Perfectly Calibrated Point Predictors and an extension of Venn-Abers calibration to the regression setup.

Strengths

- I like the interpretation of $f(X)$ as a "scalar dimension reduction". - Generalizing self-consistency to prediction sets may inspire future investigation in the broader CP community. - The idea of leveraging Venn-Abers calibration to the regression setup is a valuable contribution.

Weaknesses

- The authors do not explain, in an intuitive way, why conditioning on $f(X)$ is a good approximation of conditioning on $X$. They could describe their 1D dimensional-reduction argument better, possibly with a concrete example. Imagine $(X, Y)\in {\mathbb R}$ and $Y \sim ({\bf 1}(X<0) + 10 {\bf 1}(X<0)) {\cal N}(0, 1)$. A perfect point-predictor is $f(X) = {\rm E}(Y|X) =0$ for all $X$. In this case, conditioning on $X$ or $f(X)$ does not look equivalent. - The description of Venn-Abers calibration can be improved. I have not fully understood the meaning of these two sentences [1][2]. - The authors may explain more intuitively why they need a Perfectly Calibrated Point Prediction to compute a Self Calibrated Prediction Interval. - The proposed model is only compared with Mondrian CP. [1] *Venn-Abers calibration accounts for overfitting by widening the range of the multi-prediction in such scenarios, thus indicating greater uncertainty in the value of the perfectly calibrated point prediction.* [2] *Each point prediction in the set enjoys the same large-sample calibration guarantees for isotonic calibration.*

Questions

- Intuitively, how does conditioning on $f(X)$ help? How does this avoid the *curse of dimensionality*? What is the difference compared with conditioning on $Y$? - Is $1({Y \in C})$ the indicator of ${Y \in C}$? Why is $f$ called a *covariate shift* in (3)? How is this the same as saying that $f(X)$ is the model output? - Has prediction-conditional validity been used before? - How is $f^{X, y}$ defined? Should $f_{n+1} = ( f_n^{X_{n+1}, y}(X_{n+1}), y \in {\cal Y} ) $ be interpreted as a recursive definition? How does $f_{n+1}$ depend on $x$? - Would it be possible to compute the marginal prediction band centered on the Venn-Abers calibrated predictor (the black lines in Figure 2, if I understand the plots correctly) instead of the original one?

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

The authors should have said why they do not compare with other approximations of context-conditional validity, e.g. Conformal Quantile Regression, Ref 22 in the paper, or the Error Reweighting method [3]. [3] Papadopoulos, Harris, Alex Gammerman, and Volodya Vovk. *Normalized nonconformity measures for regression conformal prediction.* Proceedings of the IASTED International Conference on Artificial Intelligence and Applications (AIA 2008). 2008.

Summary

The paper proposes a method that jointly calibrates point predictions and provides prediction intervals with valid coverage given the point predictions. This is performed by combining two existing post-hoc processing procedures, Venn-Abers calibration and conformal prediction. The analysis of the method provides guarantees as in CP methods and a convergence rate for the conformity scoring function. The method provides a calibrated model that is piecewise constant and adapts to outcome heteroscedasticity.

Strengths

The paper is clearly written and provides a good overview of the background material on both CP and Venn-Abers calibration. The motivation for the methodology is sound. The main contribution of the paper is the theoretical analysis that combines existing results in a novel way. The algorithm is well-presented and appears relatively easy to implement.

Weaknesses

One of the main disadvantages of the proposed approach is the computational complexity, as discussed in Sec. 3.4. Due to the high computational complexity, it may not offer significant advantage over other existing methods, e.g. the Mondrian CP. It is not immediately clear from the paper what is the advantage offered over existing methods.

Questions

What is the meaning of the subscript of $f_{n+1}$ in the desiderata (line 90)? The colour (blue) of the results in Fig. 2 is not helpful. Would it be better to highlight which methods perform best? Is there a reason you are not comparing to the method in [42]? It seems to be the closes related method. The example presented in the experiment section is interesting and helpful in explaining the methodology. However, could you provide other examples of datasets or problems where your methodology would offer significant advantage over existing methods? And when would it fail in comparison to existing methods? Did your theoretical analysis offer any general insights about the class of problems that you are addressing in this work? Do you consider the theoretical analysis to be the main contribution of this work? Can the analysis be reused or easily adapted to other methods? Minor: The citations do not seem to be linked.

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper contains a discussion of the limitations (in Sec. 3.4) as well as appendix B.3. However, I believe that the limitations discussed in the Paper Checklist should be moved to the main part of the paper as it would contribute to the exposition of the methodology.

Summary

This paper introduces Self-Calibrating Conformal Prediction a novel uncertainty estimation framework combining Venn-abers calibration with conformal prediction. The output provides calibrated point predictions with associated prediction intervals with validity conditional on these model predictions.

Strengths

1. Originality: The paper presents a novel combination of Venn-Abers calibration and conformal prediction, extending Venn-Abers to regression settings. This approach to achieving both calibrated point predictions and conditionally valid prediction intervals is novel. 2. Quality: The theoretical foundations are well-developed, with clear assumptions and complete proofs provided in the appendix. The authors prove important properties of their method, including perfect calibration of the Venn-Abers multi-prediction (Theorem 4.1) and self-calibration of the prediction interval (Theorem 4.2). 3. Clarity: The paper is well-structured and clearly written. The motivation and formulation are very well done. 4. Significance: The proposed method addresses an important ML uncertainty estimation problem - providing reliable uncertainty estimates that are both calibrated and conditionally valid. This problem is important in the context of trustworthy ML.

Weaknesses

1. Limited experimental evaluation: While the MEPS dataset experiment is a nice case-study, the paper would benefit from evaluations on a wider range of datasets as is common in conformal prediction works. E.g. the widely used conformal regression datasets from CQR (https://arxiv.org/abs/1905.03222). The synthetic setups in the appendix help, but more real world datasets would be useful 2. Comparison to other conformal regression approaches: besides the baseline marginal CP, it would be helpful to see where other robust conformal approaches like CQR fit in compared to SC-CP— as they all should have good coverage and width, but it would be useful to see their calibration relative to the proposed method.

Questions

1. Can the authors provide more insight into the trade-offs between computational complexity and accuracy when using the approximations suggested for non-discrete outcomes? 2. How sensitive is the method to the choice of the initial predictor f? Are there certain types of predictors that work particularly well or poorly with SC-CP? E.g. an xgboost was used how would the performance differ for other predictors f

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have done a good job of addressing the limitations of their work. They discuss potential computational challenges for non-discrete outcomes and suggest approximations. Other limitations are briefly sprinkled throughout the paper. To improve this aspect, a suggestion is to include a dedicated "Limitations" section that consolidates and expands upon the current discussion of limitations scattered throughout the paper to make readers more aware of these factors.

thank you for your answers

I am happy with the authors' explanations and will raise my score to 5.