This paper introduces a boosted conformal procedure designed to tailor conformalized prediction intervals toward specific desired properties, such as enhanced conditional coverage or reduced interval length. We employ machine learning techniques, notably gradient boosting, to systematically improve upon a predefined conformity score function. This process is guided by carefully constructed loss functions that measure the deviation of prediction intervals from the targeted properties. The procedure operates post-training, relying solely on model predictions and without modifying the trained model (e.g., the deep network). Systematic experiments demonstrate that starting from conventional conformal methods, our boosted procedure achieves substantial improvements in reducing interval length and decreasing deviation from target conditional coverage.
Paper
Similar papers
Peer review
Summary
This paper introduces a gradient boosting-based approach to tailor a conformal score function to better satisfy desirable properties. A more general score function "family" is introduced which adheres to a specific form dependent on parameters (e.g. $\mu, \sigma$). Instead of directly using the parameter estimates obtained from model training of the predictor $f$, a post-hoc boosting stage is introduced in which the parameters are further refined leveraging additional data to minimize property-specific losses, such as conditional coverage deviation or conformal interval length. After boosting for $\tau$ iterations, the refined parameter estimates $\mu^{(\tau)}, \sigma^{(\tau)}$ can be used for down-stream conformal calibration and prediction interval generation. Results are compared to default local conformal scores (Local) and CQR on a range of regression tasks, obtaining slightly more favorable empirical properties.
Strengths
- Addressing the practicality of obtained conformal sets is a relevant problem, since results can sometimes be overly conservative to adhere to the conformal guarantees - The proposed method seems relatively lightweight and has the benefit of not requiring changes to the model weights of $f$, since the boosting derivatives are computed w.r.t. the inputs of the scoring function $E$, which are the predictions $\mu(x), \sigma(x)$ - The improvements for the local conformal score function seem substantial at times
Weaknesses
My main concerns are in regards to evaluation and practicality. The authors propose two applications of the boosting procedure to improve conditional coverage and prediction set size. For conditional coverage, the boosting loss is introduced as a relatively complicated group deviation metric leveraging a particular "contrast tree" mechanism, and in practice requires further non-trivial approximations to make it differentiable for boosting. Since the results are very strongly tied to this "contrast tree" approach, I was struggling to see how the boosting loss and conditional coverage target (Eq 12) connect in practice and how the results (e.g. Fig. 3) are more widely applicable, since the procedure is motivated as an "effective evaluation metric (for target conditional coverage) of independent interest". Is this procedure amenable to any black-box predictor $f$ and can be considered a particular gradient boosting mechanism? In general, I found the obtained improvements not overly convincing. For example, in Fig. 3 if we consider the % of data for which conditional coverage ends up being violated (above the red line) we find 33% for Local and 37% for Boosted Local. Granted, the boosting seems to distribute miscoverage somewhat more evenly, but the overall violation rate remains similar. In Table 1, boosting CQR only gives marginal improvements. It would be good to compare the results to other conformal algorithms which claim to empirically (or sometimes even theoretically, under relaxed conditions or varying interpretations of conditionality) improve conditional coverage, e.g. [1,2,3,4]. This would help clarify if the approach is more widely beneficial. Similarly, all experiments are run leveraging the same random forest and quantile network regressors -- do results translate and are amenable across different models $f$ as well? Another hurdle might be the fact that a differentiable property-specific loss needs to be defined for every target property, whose amenability may also depend in some way on the underlying model's design, and is fixed. Thus, tackling a more flexible scoring function comes at the cost of introducing another (fixed) functional to optimize over, with required approximations. On another note, I was somewhat struggling to understand how the boosting procedure connects to the overall conformal procedure. In Fig. 1 it is suggested that boosting is an independent step inbetween model training and conformalization, but in the experiment description it seems that boosting leverages existing datasets (namely training data). Should I understand that the boosting dataset $\mathcal{D}$ is a separate set? If so, how do its sampling requirements (e.g. exchangeability) connect to calibration and test data. If not, should I understand that the boosting step actually equates to an iterative conformal calibration procedure leveraging $\mathcal{D}_{cal}$? Since the boosting objectives (Eq. 14, 17) actually require computing conformal sets and quantiles. Thank you for clarifying. Other comments: - I believe the $j$ index in the 3rd final line of Algorithm 1 should be omitted, since the final boosted score functions are not taken w.r.t. a particular fold. - Fig. 4 is missing a legend, making it hard to interpret - The use of the term "Power Loss" seems odd, since the goal is to reduce prediction set size, and from my understanding a smaller prediction set/interval will equate higher power, since the permutation testing interpretation of the null hypothesis is on inclusion in the set. - Theorem 5.1 does not seem like a theorem but more like a remark to me, since it is merely suggesting that the "generalized" score family may contain the optimal oracle solution, but in no way relates any optimality to the practically obtained solutions via the boosting procedure. - It would be beneficial to use hyper-referencing for Equations and the Algorithm throughout the paper. References - [1] Romano, Yaniv, et al. "With malice toward none: Assessing uncertainty via equalized coverage." Harvard Data Science Review 2.2 (2020): 4. - [2] Gibbs, Isaac, John J. Cherian, and Emmanuel J. Candès. "Conformal prediction with conditional guarantees." arXiv preprint arXiv:2305.12616 (2023). - [3] Sesia, Matteo, and Yaniv Romano. "Conformal prediction using conditional histograms." Advances in Neural Information Processing Systems 34 (2021): 6304-6315. - [4] Jung, Christopher, et al. "Batch multivalid conformal prediction." arXiv preprint arXiv:2209.15145 (2022).
Questions
- Could you please comment on some of the raised points above, such as on clarifying the data used by the boosting procedure, the applicability of the "contrast tree" method, the practicality of defining new optimization objectives, or the distinction to other existing methods aiming for conditional coverage. - The approach to obtain a differentiable loss objective from Eq. 14 seems quite involved. Could you comment on the motivation of leveraging "contrast trees" and the tools used to go from Eq. 14 to a differentiable loss? - Are there any results that can be made on how the boosting procedure affects down-stream conformal coverage guarantees, or statements of optimality or improvement with regards to the obtained "boosted" conformity scores over the default ($0$-th iteration) conformity scores?
Rating
6
Confidence
4
Soundness
2
Presentation
3
Contribution
2
Limitations
I did not find the limitations of the method thoroughly discussed anywhere, and I believe the distinction to other related works could be improved. There are some comments made in the discussion on improvements, as well as L218-L222 on not guaranteeing an optimal solution.
Summary
This paper introduces a methodology for learning a conformal score function after training. Notably, the proposed method does not require model retraining, and instead learns the score function using trained model predictions using a cross-validation approach on the training data.
Strengths
**Originality:** Although the paper is in the same vein as works such as ConfTr [1], the fact that it can work directly on top of trained predictions on the same data is a clever formulation and a nontrivial benefit. **Quality:** Overall, the paper is technically sound; I have no concerns regarding the theory. My reservations regarding the empirical results are discussed under weaknesses. **Clarity:** The paper is well-written and all necessary ideas regarding conformal prediction for understanding the results in the paper are discussed appropriately. **Significance:** The idea of learning a conformal score function after training (using the same data) but before calibration is, to my knowledge, a novel formulation and worth building on. [1] https://arxiv.org/abs/2110.09192
Weaknesses
**Marginal empirical benefit.** My main reservation with the proposed method is the seemingly marginal benefit it provides over CQR for conditional coverage, as shown in Table 1. This is especially relevant for large scale settings where the overhead of the boosting/CV stage is going to be highly nontrivial. Additionally, conditional coverage here is evaluated using contrast trees, which are baked into the objective of the boosting approach. It would be interesting to see if the improvements hold under, for example, just group-conditional coverage. **Baseline score functions.** The baseline score functions as described in Appendix A.8 are relatively weak, untuned models. While I do not expect an expansive evaluation of models to be within the scope of this work, it would be good to know how sensitive the improvements in the paper are to the choice of baseline score model. Overall, while I have reservations regarding the empirical results, I do find the formulation of learning the conformal score in this paper useful. As a result, I lean accept.
Questions
- The boosting stage uses the same data on which the baseline score is trained. I'm curious whether the authors also considered splitting off some data for this stage (instead of doing the CV approach)? Although this would be less sample efficient, it seems like it may have greater benefit than reusing data for more powerful baseline score models. - It seems to me that the choice of boosting here for improving the baseline score could be replaced by some alternative means of mapping trained model predictions to the necessary mean/variance parameters for the conformal prediction intervals, and I'm wondering whether the authors considered such alternatives?
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Although not in a separate section, limitations are adequately discussed.
Summary
The paper proposes to utilize the training data and gradient boosting (of the model's predictions) to optimize loss functions related to conformal prediction interval length and conditional coverage with respect to the score function. The optimized score function is then used for "plain" CP usage, via calibration set.
Strengths
- The approach is novel and significantly differs from other methods that utilize training data to improve CP performance. - The authors derive asymptotic theory for the sufficient expressiveness of the approach. - Experiments demonstrate that the approach consistently boosts the CP metric associated with the designed loss (interval length or conditional coverage).
Weaknesses
- There is no discussion and investigation on the impact of optimizing the score for conditional coverage on the interval length and vise versa. In practice, both can be important for a user. - Even though there is theory of the "asymptotic expressiveness" of the approach, the method in practice is based on differentiable approximations and greedy optimization that do not guarantee optimality.
Questions
- What can you say about the computational complexity of the boosting procedure? - In practice, both conditional coverage and the interval length are likely to be important for a user. Therefore, when you optimize the score for better conditional coverage report also the effect on the interval length. Similarly, when you optimize the score for better interval length report also the effect on the conditional coverage. - Regarding the previous comment, when discussing these two properties it is worth mentioning the work: Dabah, L., & Tirer, T. (2024). "On Temperature Scaling and Conformal Prediction of Deep Classifiers". arXiv preprint arXiv:2402.05806. that shows a simple way to trade between them in CP methods for classification. - While I am aware that your approach do not require access to the model's parameters, it would be insightful to compare the approach (even at some of the settings) to the approach of [26], namely optimizing the model's parameters to minimize a loss that includes the term that you design.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
n/a
Summary
The paper resents a novel method to enhance conformal prediction intervals using gradient boosting. The proposed method focuses on improving specific properties such as conditional coverage and interval length. The key idea is to iteratively refine a predefined conformity score function via boosting, guided by a loss function designed to measure deviations from the desired properties. Experiments demonstrate that starting from conventional conformal methods, the boosted procedure significantly reduces interval lengths and better aligns with target conditional coverage.
Strengths
1. Post-Training Enhancement: The boosting process is applied post-training, meaning it doesn't require modifications to the original model, making it applicable to a wide range of pre-trained models. 2. Flexibility: The approach can be tailored to optimize various properties beyond interval length and conditional coverage, such as fairness or other specific criteria. 3. Robustness: The use of cross-validation to select the number of boosting rounds helps prevent overfitting and ensures the method's robustness across different datasets and applications.
Weaknesses
1. A comprehensive description of the experiment setting is necessary. Since the method is post-training, it is crucial to explain how the training samples used for post-training differ from those used for the base model. Additionally, it is important to address potential outcomes if a distribution shift occurs. If the data used for post-training is different from the one use for calibration and testing, can it still works well? (or if the data used for (posted-training,calibration, testing) is different from the one used for base model training, can it still achieves to improve CP?) 2. For experiment presentation, why the tables contain only conditional coverage and size solely? I mean, to convincingly demonstrate the method's effectiveness, it is essential to show that the method can improve interval length without significantly compromising conditional coverage, or improve conditional coverage without significantly increasing interval length.(rather than showcase them solely) 3. The introduction of a boosting step increases the computational burden, potentially requiring significant resources and time, especially for large datasets. An analyze for time complexity is required for better illustration.
Questions
1. Can your method work well with different error rates $\alpha$? 2. If the post-training set is small, can your method still perform effectively?
Rating
6
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Limitations
Please refer to the weakness.
Thank you for the thorough response, in particular the provided example. I plan to keep my score as I was already favoring acceptance, and in my view raising the score would require more substantive experiments that extend beyond a minor revision.
Thank you for the thorough response, in particular the supplementary experiments, I raise my rating to 6.
Thank you for your answer and clarifications. A few more comments: > Please refer to point 3 in the global rebuttal. It would be nice to see more experimental validation to support your claims on leveraging your boosting procedure as a general tool to optimize for desired properties, and your suggestion for FDR control seems interesting. Naturally, this seems out of scope for the current work but perhaps could be added to future work / discussion. > Those works measure group conditional coverage based on user-specified groups. However, relying on predefined groups may not always be ideal This is a valid point, but a counterargument can be made for the fact that when we work in the group-conditional setting we often either know (or even define) interesting group partitions and may only be interested in these particular groups or a subset thereof, whereas coverage imbalance for other partitions is amenable (e.g. in the fairness case). Your contrast tree approach seems to perform its own partitioning in feature space that can (i) provide group partitions that are not really meaningful or hard to interpret, and (ii) end up suboptimal for particular groups of interest. Thus I still believe it would be useful to compare to such approaches in an experimental setting where groups are known a priori, and demonstrate that the contrast tree approach recovers meaningful partitions. At the very least, a clearer separation from such works should be done. > We agree with the reviewer that optimizing a targeted property requires a customized objective. I would suggest to stress this limitation for reader clarity, since I do not see it explicitly mentioned anywhere. This seems like a key modelling design step that can prove challenging for anyone wanting to boost for alternative objectives. In particular, it is unclear how the approximations made to obtain a differentiable objective impact the fundamental conformal prediction outcomes (e.g., validity and set sizes). > Please refer to Figure 2 for an illustrative schematic drawing. We shall work to explain this more clearly in the revised manuscript. Thank you for clarifying. I agree that this needs clarification, because there is no direct connection established between Fig. 1 and 2, and they seem to suggest different conceptual approaches (post-hoc vs. leveraging training data). Relating to this and my rebuttal (*If so, how do its sampling requirements (e.g. exchangeability) connect to calibration and test data.*), it seems that leveraging the training data during boosting may impose an additional assumption on the training distribution being exchangeable with the calibration and test distributions. If so, this is not line with *"the usual exchangeability assumption"*. It seems like this data reuse during boosting is a key step that complicates the possibility to provide theoretical guarantees of some form on your procedure, would you agree? > We agree with the reviewer that a smaller prediction set or interval corresponds to higher power. In the manuscript, by "power loss," we intended to convey "loss of power," meaning that when the interval length is larger, we lose power. If the reviewer finds this phrasing misleading, we will rephrase it in the revised manuscript. I do find it somewhat confusing, especially given that the multiple testing interpretation is not touched upon anywhere else in the paper, and would suggest renaming or omitting this. > We agree with the reviewer that Theorem 5.1 only proves that the search space contains the optimal solution, but does not guarantee the optimality of the numerical boosting procedure. In my opinion this makes Thm 5.1 a fairly weak statement, and I would suggest renaming it into a "remark" or "proposition". In practice, this does not provide me with anything of real value. Since it follows primarily from your definition of generalized score families, perhaps Thm 5.1 as well as Thm. A.2 can even be rephrased as results following from a theoretical statement on "inclusion of the optimal solution in the search space of generalized score families" under sec 3. Looking forward to your reply.
Replying to Official Comment by Reviewer 3gAf
We thank the reviewer for taking the time to review our rebuttal and provide additional helpful comments. > It would be nice to see more experimental validation to support your claims on leveraging your boosting procedure as a general tool to optimize for desired properties, and your suggestion for FDR control seems interesting. Naturally, this seems out of scope for the current work but perhaps could be added to future work / discussion. We shall include this in the discussion of additional objectives in Section A.1. > Thus I still believe it would be useful to compare to such approaches in an experimental setting where groups are known a priori, and demonstrate that the contrast tree approach recovers meaningful partitions. At the very least, a clearer separation from such works should be done. We agree with the reviewer that there are situations where groups defined based on domain knowledge a priori are of interest. In response, we further conduct the following experiments. As discussed in Section A.1, we can readily extend our boosting procedure to the group conditional coverage objective. As a result, we run the boosting procedure on the Meps-19 dataset divided into 4 groups of non-white female, non-white male, white female and white male. We observe a significant improvement (54%) for the Local baseline, and a moderate improvement (15.6%) for the CQR baseline. | | Marginal Coverage | Max within group deviation from target coverage rate | Average length | :---------------- | :----: | :----: | :----: | | Local | 89.6% | 8.0% | 2.16 | Boosted Local | 89.8% | 3.7% | 2.67 | CQR | 89.4% | 4.2% | 3.26 | Boosted CQR | 89.7% | 3.5% | 3.21 >I would suggest to stress this limitation for reader clarity, since I do not see it explicitly mentioned anywhere. This seems like a key modelling design step that can prove challenging for anyone wanting to boost for alternative objectives. We shall discuss this limitation in our revised manuscript. > In particular, it is unclear how the approximations made to obtain a differentiable objective impact the fundamental conformal prediction outcomes (e.g., validity and set sizes). We agree with the reviewer that in general, using an objective that does not directly target set size may lead to undesirable (or more desirable, as shown in the comparison between CQR and Boosted CQR in the experiment above, where the target was group conditional coverage) set sizes. As a result, we would recommend that the user incorporate all the targeted properties when customizing the optimization objective. We shall defer the discussion on validity to the reviewer’s next question. > Relating to this and my rebuttal (If so, how do its sampling requirements (e.g. exchangeability) connect to calibration and test data.), it seems that leveraging the training data during boosting may impose an additional assumption on the training distribution being exchangeable with the calibration and test distributions. If so, this is not line with "the usual exchangeability assumption". It seems like this data reuse during boosting is a key step that complicates the possibility to provide theoretical guarantees of some form on your procedure, would you agree? We respectfully disagree with the reviewer on this point. The guarantee of marginal coverage validity for our boosted procedure does not require additional assumptions beyond the exchangeability assumption already made for the split conformal procedure. Whether or not boosting is applied, the training set produces a fitted or boosted conformity score based on the training data. As long as the calibration set and test set remain exchangeable, the original proof of validity still holds. This is also one of the reasons we chose to use the training set for boosting. We hope this clarifies our approach and rationale. > I do find it somewhat confusing, especially given that the multiple testing interpretation is not touched upon anywhere else in the paper, and would suggest renaming or omitting this. We thank the reviewer for clarifying. We shall omit this in the revised manuscript. > In my opinion this makes Thm 5.1 a fairly weak statement, and I would suggest renaming it into a "remark" or "proposition". We shall rename it as a proposition in the revised manuscript.
Thank you for your response. > As long as the calibration set and test set remain exchangeable, the original proof of validity still holds. This is also one of the reasons we chose to use the training set for boosting. We hope this clarifies our approach and rationale. I don't see this explicitly mentioned in the paper, so I would suggest to make this more apparent. If this is the case, the fact itself could be formalized further in a proposition that extends the guarantee of split CP to incorporate the boosting stage. This seems, in my opinion, like a much more practical statement to formalize than the current Thm 5.1. Overall, while I am still not entirely sold on the contrast tree approach and on the intricacies of your guarantees connecting to approximations of your boosting objectives, I think there are some nice ideas that could benefit practitioners in the CP field, in particular leveraging training data but moving away from the conformal training procedures which I find quite subpar. Thus I am raising my score and looking forward to the revised version incorporating your stated improvements.
We sincerely thank the reviewer for their helpful suggestions to improve our paper. > I don't see this explicitly mentioned in the paper, so I would suggest to make this more apparent. If this is the case, the fact itself could be formalized further in a proposition that extends the guarantee of split CP to incorporate the boosting stage. This seems, in my opinion, like a much more practical statement to formalize than the current Thm 5.1. We agree with the reviewer that this is worth clarifying. We shall include this as a proposition in the revised manuscript.
I thank the authors for their response and keep the score that I gave (which already supports acceptance). As discussed, I am keen to see in the next version the effect of the approach on the trade-off between conditional coverage and interval length, which is currently not presented.
Decision
Accept (poster)