Conformal Prediction for Dose-Response Models with Continuous Treatments

Response to review

Thank you for your review. We addressed your questions and addressed weaknesses as follows. **Methodological Contribution & Novelty:** We acknowledge that the method builds on prior work, including the works of Tibshirani et al. (2019) and Lei and Candes (2021). However, our work focuses on the conditional average dose-response function (CADRF) compared to CATE estimation, which quantifies the causal effect, while we aim to provide dose-response curves within the potential outcome framework. Additionally, we generalize the work of Lei and Candes, which considers counterfactual inference in the binary treatment setting, to the continuous treatment setting. In the appendix (Appendix A, revised paper), we added the theoretical coverage guarantees of our proposed approach, together with a discussion of the desired coverage guarantees. We additionally show here that the same coverage guarantee as in Lei and Candes for binary treatment is impossible for continuous treatment without creating trivial intervals. Therefore, we introduce the context of a shift in the treatment distribution to the interventional distribution. For a more in-depth discussion see Appendix A (see revised paper). Additionally, we included recent work by Schroder et al., as it is closely related to our approach, though published contemporaneously (per ICLR 2025 guidelines). We included this for transparency and comparison. Additionally, to our knowledge, Schroder et al.'s current approach would be computationally infeasible in our simulation setting (time-wise), which reduces practical utility. **Theoretical Guarantee & Validation:** We have added a theoretical guarantee in Appendix A to support our results on synthetic datasets. In the Experiments section, we clarified that models like Gaussian Processes, Conformal Prediction, CatBoost with Uncertainty, and Local WCP serve as baseline models for comparison. These represent naive approaches to uncertainty quantification for dose-response curves. We are constrained by the lack of counterfactual observations to evaluate our method regarding real-world data. Hence, we added the theoretical guarantee. **Extensions & Applications:** We also expanded Appendix D to discuss potential applications and extensions, further highlighting the method's utility in real-world contexts, including clinical trials and other decision-making domains.

Response to review

We would like to thank the reviewer for their thoughtful feedback and comments. We addressed the questions and weaknesses as follows. **Theoretical Contributions & Methodology:** While the methodology builds on existing CP and propensity score techniques, we believe the novel contribution lies in its application to dose-response modelling with uncertainty quantification for counterfactual treatments. We have included potential extensions and further applications of the method in Appendix D to highlight its broader applicability. Additionally, we present a more general theoretical framework for counterfactual inference and WCP in the revised paper (see Appendix A, revised paper). **Real-World Data & Practical Utility and Application Beyond Healthcare:** We agree that real-world validation is crucial. However, real-world evaluation is challenging due to the need for counterfactual observations of all possible treatments for each individual. In practice, real-world data typically only observes a single treatment per individual. Therefore, coverage guarantees cannot be measured without counterfactuals. However, in Appendix A we added a theoretical guarantee to support our coverage guarantee claims, to compensate for this lack of real-world application. However, we discuss potential real-world applications in Appendix D, where the method can be applied to fields such as clinical trials, preventive maintenance, and sales. The interventional distribution assumptions about covariate shift remain similar across domains, as the goal is to consider all treatments equally in decision-making. If covariate shifts between train and test in the features $X$ are measurable or known, they can be incorporated into the method’s weights. **Scalability & Computational Overhead:** We have added a computational overhead analysis in Appendix C. The overhead scales linearly with the number of treatments evaluated for Local Propensity WCP (the most computationally intensive). Calibrations can be performed beforehand, and the primary complexity depends on the base learner. Treatment evaluations can be parallelized in real-time settings, allowing for inference in the second-to-millisecond range, depending on the base learner's inference time. **Robustness of Propensity Estimation:** We have included a robustness analysis in Appendix A, which discusses the relationship between errors in propensity estimates and the method’s coverage guarantees. The main limitation of KDE is the sensitivity to kernel choice and hyperparameter tuning which must be considered when implementing our version for propensity estimation. However, KDE is primarily used for smoothing and generating continuous density functions. Our Local Propensity WCP method is not limited to the CPS with KDE approach.

Response to review

Thank you for your review. Below, we address each of your concerns in detail. **Theoretical Guarantee:** We have added a theoretical finite-sample coverage guarantee, with a lower and upper bound, for any interventional distribution method using both oracle and estimated propensity function in Appendix A (see revised paper). This formalizes the method’s performance and provides the necessary theoretical foundation. **Illustrations and Algorithm:** To clarify the method, we have included pseudocode for both Global and Local Propensity WCP in Appendix C. This should help readers understand the fitting, calibration, and inference processes. We believe this addition also provides a clearer illustration of the method's workflow. This also clearly illustrates now that CPS is used to estimate the propensity function in combination with KDE, which was also already mentioned in Section 5.2; however, it was not that clear; hence we rephrased it (see line 384, revised paper). **Assumptions:** Regarding the assumption of no covariate shift between training and test data, we agree this is a simplifying assumption. Appendix D discusses potential extensions of both Global and Local Propensity WCP that account for covariate shifts in $X$. While this assumption was used to simplify the derivations, the method can handle covariate shifts in practice if we account for them in the weigths. If covariate shifts are measurable, they can be easily incorporated into both Global and Local Propensity WCP, and we have updated the methods section to include a reference to this. Regarding the uniform interventional distribution assumption, the uniform distribution comes from a decision-making standpoint when an intervention has yet to be performed, and we want to evaluate every treatment value equally. Hence we want an unbiased uncertainty quantification where we aim to evaluate all treatment values equally, as in a clinical trial. We added more nuance to the methodology section to clarify this. The coverage guarantee in Appendix A (see revised paper) is also general, allowing the use of any interventional distribution; our proposed interventional distributions are also discussed and translated into the general theoretical framework. **Numerical Experiments and Method Comparison:** We have updated the experiments section to clarify the differences between the various methods. Specifically: - **WCP Local Propensity** and **WCP Global Propensity** are our contributions, with **WCP Local Propensity** being the primary focus of this work. - **WCP Global** uses the global weights $w_{g,p}$, while **WCP Local** uses local weights $w_{l,p}$ as outlined in the methodology. - The other methods, such as CP, Gaussian Processes, CatBoost with Uncertainty, and Local WCP, serve as baseline comparisons. **Real-World Data and Application:** We agree that real-world application is important. However, evaluating the method with real-world data is challenging due to the inherent limitations of only observing a single treatment for each individual. Evaluating our method requires counterfactuals for all possible treatments for a single individual, which is not feasible with real-world data where only one treatment is observed. Thus, we focus on synthetic data to evaluate coverage guarantees. While we recognize this limitation, real-world validation is an area for future exploration; therefore, we added a discussion on the potential applications to Appendix D to cover this as well.

Response to review

We would like to thank the reviewer for their thoughtful feedback and comments. We addressed the questions and weaknesses as follows. We agree that real-world validation is crucial for demonstrating the practical applicability of our method. However, evaluating the model on real-world data would not verify the necessary coverage guarantees, as this would require counterfactual outcomes for every possible treatment value and every sample, which is typically unavailable in real-world settings. In practice, we only observe a single treatment per sample, while our method is designed to quantify uncertainty across all counterfactuals. Therefore, we evaluate our method using synthetic data, where the true counterfactuals are known, to ensure that the method works as expected. To address the need for coverage guarantees, we have included theoretical coverage guarantees in Appendix A (see revised paper) that formalize the coverage for all counterfactual treatments and give a lower and upper bound for the coverage when using both the oracle and estimated generalized propensity function. While synthetic data is helpful for this purpose, we recognize that real-world applications are the ultimate goal. To that end, we have expanded the appendix with a discussion of potential applications for our method, including clinical trials, preventive maintenance, and sales. One challenge in applying the method to real-world data is the lack of overlap in treatment distributions, especially in areas like drug dosing, where doses are determined by predefined treatment protocols, limiting the range of treatments across samples. Our method’s uncertainty quantification is intended to assist decision-making by accounting for counterfactuals. However, translating this uncertainty into actionable, optimal decisions remains an area for future work, particularly when uncertainty quantification could provide complete distributions, such as using CPS. For that domain, clinical utility, decision functions, or offline reward/value/outcome can be explored in future work.

Response to review

We would like to thank the reviewer for their thoughtful feedback and comments. We addressed the questions and weaknesses as follows. **Error of Estimated CADRF:** We acknowledge that the error of the estimated CADRF was only shown indirectly in Figure 2 for one of the experimental setups. In response, we have added RMSE results for all experiments and treatment values in Appendix F. However, as our method is model-agnostic, the RMSE could be further improved by using more suitable models and tuning hyperparameters. **Conservative Intervals:** The conservativeness is entirely determined by the number of samples in the calibration set (in the case of split-WCP) and how well-behaved the likelihood ratio is; we included a more theoretical discussion of the upper bound of the coverage in Appendix A (see revised paper). If the prediction intervals become infinite, this indicates that in these regions, there is not enough data support (i.e., lack of overlap) for this sample to provide a counterfactual prediction. Thus, the model cannot be trusted here. For example, assume that the overlap or positivity assumption is violated, i.e., $\frac{d\tilde{P}{T|X}}{dP_{T|X}} = \infty$ in terms of the interventional distribution, this will result in the trivial interval $(-\infty, \infty)$, since $w(X_i)=0, \forall i \in [1,...,n]$ and $w(X_{n+1})=\infty$ resulting in $p^w_i(X_{n+1})=0, \forall i \in [1,...,n]$ and $p^w_{n+1} = 1$. The reason for quite conservative coverage in the experiments is that the covariate distribution $P_X$ remains fixed while we shift the treatment distribution. This can result in sharp likelihood ratios, reducing the effective sample rate. However, in places with enough overlap, the empirical coverage is close to the target coverage, which aligns with the theoretical results in Appendix A. To achieve empirical coverage closer to the target coverage, one could increase the calibration samples; another approach would be to use a smoothing term drawn from the uniform distribution, allowing exact coverage guarantees under the Oracle propensity function. However, this would result in non-deterministic prediction intervals.