Zero-shot causal learning

Predicting how different interventions will causally affect a specific individual is important in a variety of domains such as personalized medicine, public policy, and online marketing. There are a large number of methods to predict the effect of an existing intervention based on historical data from individuals who received it. However, in many settings it is important to predict the effects of novel interventions (e.g., a newly invented drug), which these methods do not address. Here, we consider zero-shot causal learning: predicting the personalized effects of a novel intervention. We propose CaML, a causal meta-learning framework which formulates the personalized prediction of each intervention's effect as a task. CaML trains a single meta-model across thousands of tasks, each constructed by sampling an intervention, its recipients, and its nonrecipients. By leveraging both intervention information (e.g., a drug's attributes) and individual features~(e.g., a patient's history), CaML is able to predict the personalized effects of novel interventions that do not exist at the time of training. Experimental results on real world datasets in large-scale medical claims and cell-line perturbations demonstrate the effectiveness of our approach. Most strikingly, \method's zero-shot predictions outperform even strong baselines trained directly on data from the test interventions.

Paper

References (100)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer fcYE7/10 · confidence 4/52023-06-17

Summary

In this paper, authors have proposed causal meta-learning framework that trains a single model across several intervations and estimate CATE for novel interventions at inference time which were not present at the training time. They frame CATE estimation for each intervention as a separate meta-learning task. The proposed framework is evaulated on two real world large-scale datasets as medical claims and cell-line perturbations, which show the effectiveness of the proposed framework.

Strengths

- A new meta-learning framework, called CaML, is proposed to estimate treatment effect for novel interventions by considering each intervention as a separate task. This is simple but interesting formulation. - CaML is evaluated on two real world large-scale datasets as medical claims and cell-line perturbations, which show the effectiveness of the proposed framework - Theoretical results are provided to bound the prediction error (I have not verified these results). - Code is provided with the paper (I have not checked it).

Weaknesses

- The idea seemed simple and interesting but I found it hard to follow, especially the results. Why are the treatment effect estimates first calculated and used as kind of lables, and why not direclty learn the model? The selected CATE learner (RA-Learner) can create biased estimates. There are typos which also make some sentences hand to interpret. Please revise accordingly. In standard ML, use of validation dataset makes sense but what does validation tasks/interventions mean? What do you mean by 'Given m outcomes of interest'? Does that mean the interventions have multiple outcomes discrete/continous?

Questions

- What is the ratio of treated and untreated while preparing meta-datasets? - CaML comes in different variations, like CaML - w/o RA-learner, etc -- if you could expalin all those at one place with other baselines then that would be easy to follow. - Why very narrow range is considered for u in [0.999,0.998,0.995,0.99]?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

1 poor

Contribution

2 fair

Limitations

Yes

Reviewer o8uy7/10 · confidence 3/52023-06-28

Summary

The paper proposes a zero short cate estimation algorithm. Building natural experiments sub datasets and estimating pseudo outcomes the paper attempts at estimating the cate of unseen interventions

Strengths

- interesting premise - interesting and novel method - adequate review of related works - good experimentation and discussion

Weaknesses

- unclear why the pseudo outcome target \tau is unbiased - unclear how the algorithm would fair if the training data was biased in the prevalence of specific interventions . For example if the available data only contains a small subset of possible interventions it is possible that the generalisation of the model is not adequate to span the entire set of possible interventions

Questions

- if the authors could address the latter point in the weakness section

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- adequate discussion of limitations and impact

Reviewer qej57/10 · confidence 4/52023-07-05

Summary

The paper presents a novel framework for zero-shot causal learning, which is the problem of predicting the personalized effects of a novel intervention that has no historical data. The framework, called CaML (Causal Meta-learning), formulates the prediction of each intervention's effect as a meta-learning task, and trains a single meta-model across thousands of tasks, each constructed by sampling an intervention. The paper demonstrates the effectiveness of CaML on two real-world datasets in large-scale medical claims and cell-line perturbations, and shows that CaML can outperform existing methods that are either trained on the test interventions or capable of zero-shot generalization. The paper also provides a theoretical analysis of CaML's zero-shot generalization bound. The paper contributes to the field of causal inference by unlocking zero-shot capability for many recently developed CATE estimation methods that were previously restricted to single interventions.

Strengths

The paper presents an original approach to addressing the challenging problem of zero-shot causal learning, which has important applications in various domains including personalized medicine, public policy, and online marketing. The paper's framework is comprehensive and rigorous, and its effectiveness is demonstrated through experiments on two real-world datasets in large-scale medical claims and cell-line perturbations. Additionally, the paper unlocks the zero-shot capability for many recently developed CATE estimation methods, which were previously restricted to single interventions, and provides a promising direction for future research in causal inference.

Weaknesses

While the paper provides a thorough evaluation of its framework using metrics such as RATE and PEHE, future research should explore additional evaluation methods and metrics that can provide a more comprehensive understanding of the performance of zero-shot causal learning methods. Additionally, the paper's section on societal impacts could benefit from a more detailed discussion of the potential risks and limitations of the proposed approach. For instance, the risk of underrepresentation of underserved communities in the training data could lead to biased results, which could ultimately have negative societal impacts. Therefore, future research should address these limitations and consider approaches to mitigate potential risks associated with the proposed framework.

Questions

Above weaknesses part

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Above weaknesses part

Reviewer uxCL7/10 · confidence 3/52023-07-06

Summary

In this paper, the authors propose a new Meta Learning algorithm called CaML - to predict the personalized effects of a novel intervention. The idea is neat, and its applicability seems reasonable given the empirical results.

Strengths

- The paper's algorithm seems to perform well for novel settings. - The PAC-like theorem is a nice addition Based on the response I have upgraded my score to Accept

Weaknesses

- I'd like to see a theorem like the one in https://arxiv.org/pdf/2112.07602.pdf (Lemma 1 or Lemma 2), which would validate unbiasedness in different settings (assuming that pre-treatment covariates are used) - A more complex set of theorems (which requires extensions that are more complex) would be ones where the distribution of the actual meta-datasets are drifting. I think even then strong results can be obtained, which would allow the use of CaML even in settings where the novel interventions are drawn from different distributions. These might not be feasible for this paper but maybe worth addressing

Questions

- Are there any other guarantees you can make on the out of sample error without a theorem like Lemma 1 above? - What covariates can be used?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The authors address some limitations - I have outlined others in the "Weaknesses" section above

Reviewer fcYE2023-08-10

Thank you for your responses to my comments. In your rebuttal, you are using the term modelling 'potential outcomes' and 'pseudo-outcome' approach. It took me a while to understand the meaning of former. Don't you think, calling the former as 'Factual outcome' modelling will be more clear, especially when talking in context of pseudo-outcome modelling? If you talk only about potential outcome modelling then one can understand it from Rubin's PO framework. But 'potential outcomes' and 'pseudo-outcome' terms together creates confusion. Earlier, my only comment was on clarity of the paper. Authors have promised to revise the paper to add more clarity, as discussed above. So, I am assuming that will improve it and it will be easy to follow. I have one more query. I apologise, I did not ask it earlier. In this framework, which parts are relevant to causal literature. I mean, you created pseudo-outcomes using RA-Learner to create a dataset and then it becomes standard ML. So, as per my understanding, formulating the causal problem into zero-shot learning is the only novel part for the causal inference literature (and real-world application). Right? Because creating pseudo labels as well as the meta-learning part are existing techniques. Please clarify the novelty of the work to causal literature. I have also skimmed through comments of other reviewers and I did not find any major concern from other reviewers.

Authorsrebuttal2023-08-12

Thank you for your thoughtful comments, feedback, and continued engagement with our paper! > In your rebuttal, you are using the term modelling 'potential outcomes' and 'pseudo-outcome' approach. It took me a while to understand the meaning of former. Don't you think, calling the former as 'Factual outcome' modelling will be more clear, especially when talking in context of pseudo-outcome modelling? If you talk only about potential outcome modelling then one can understand it from Rubin's PO framework. But 'potential outcomes' and 'pseudo-outcome' terms together creates confusion. Earlier, my only comment was on clarity of the paper. Authors have promised to revise the paper to add more clarity, as discussed above. So, I am assuming that will improve it and it will be easy to follow. We agree that the similarity between the terminology “potential outcomes” and “pseudo-outcome” may lead to confusion with readers. The problem with referring to modeling potential outcomes as “factual outcome modeling” is that while the models are trained on factual data, they are then used to predict counterfactuals (i.e., we want to model Y(0) for treated patients, and Y(w) for control patients). In our revision, we will instead refer to the first approach as “modeling Y(w) and Y(0)”. We only refer to this approach in two places in the paper (Line 137, Line 259) as it is a mere baseline, and hence we believe it is OK to use this more verbose description to ensure clarity. > I have one more query. I apologise, I did not ask it earlier. In this framework, which parts are relevant to causal literature. (...) Please clarify the novelty of the work to causal literature. I have also skimmed through comments of other reviewers and I did not find any major concern from other reviewers. Thank you for raising this additional question. We take this as helpful feedback that we should more explicitly enumerate our contributions in our paper. We will revise our paper to do so, and to highlight the following contributions: **(1)** Our first contribution is to formulate zero-shot CATE estimation. This problem is challenging (no training data for test interventions) and omnipresent: in many real-world settings, the overwhelming majority of possible interventions are unobserved. For instance, there are ~166.4 billion possible small molecules for drugs [1], whereas it is only possible to study a small number of drug candidates in clinical trials each year. Similarly, in online marketing, there is an infinite search space of possible advertisements, and a limited number of users to run A/B tests on. Achieving zero-shot CATE estimation will thus enable personalized treatment design across many domains. **(2)** Second, our paper represents a milestone for the CATE methods literature by intentionally breaking evaluation conventions. Because treatment effects are unobserved, previous papers which introduced new CATE estimation methods typically relied on synthetic and semi-synthetic datasets [2, 3, 4]. However, this overreliance on simulated data has made it difficult to ascertain if there is actual progress that can benefit real-world applications [5]. In contrast, our study evaluates on two large-scale real-world datasets featuring millions of patients (Claims) and thousands of interventions (LINCS). The evaluation utilizes recent advancements: the emergence of biological cell perturbation data (LINCS) enabling observation of counterfactual ground truth by cloning a cell-line, and the RATE metric [6] which allows for evaluation on observational data (Claims). Our empirical evaluation yields several insights which we elaborate on in Section 6.3: (a) CaML’s zero-shot performance outperforms many prior methods trained directly on the test interventions (b) CaML is capable of higher-order generalization: trained on single interventions, it can can zero-shot generalize to pairs of (both novel) interventions (c) most methods besides CaML perform worse than simple zero-shot baselines (d) zero-shot CATE estimation can inform clinical decision-making, by identifying patients which are vulnerable to a life-threatening drug side effect (Claims eval).

Authorsrebuttal2023-08-12

continuation of response to contributions question

> pseudo labels as well as the meta-learning part are existing techniques **(3)** Successfully translating ideas from one ML subfield to another, and discovering that they unlock entirely new capabilities, is non-trivial and a contribution in its own right. For example, the Transformer architecture was originally introduced in 2017 for NLP. It wasn't until 2020 that its impact for computer vision was recognized with the landmark Vision Transformer (ViT) paper [7]. Similarly, progress in the subfield of Graph Neural Networks was sparked by the discovery that word embeddings (word2vec) can be extended to embed nodes in a graph (node2vec) [8]. In the same spirit, developing a framework that allows for pseudo-outcome labels and meta-learning to be (successfully) applied to zero-shot CATE estimation requires careful design considerations. *This is because pseudo-outcome labels and meta-learning cannot be applied to zero-shot CATE estimation out-of-the-box.* Our key insight in designing the CaML framework is that, rather than training on the raw (W, X, Y) triplets, we can synthesize natural experiments from the data to construct a distinct CATE estimation task for each training intervention. This allows us to (a) unlock zero-shot capability for many recently developed pseudo-outcome estimators which were previously restricted to studying single interventions in isolation (b) leverage meta-learning algorithms (i.e. Reptile) for meta-model training, which requires task-structured data. We discover in our experiments that both of these factors contribute to CaML’s strong empirical performance and allows it to outperform all baselines and existing methods. **(4)** Finally, our theoretical contribution is to bound the zero-shot CATE prediction error as a function of several key parameters (e.g. how informative intervention information $w$ is of the true treatment effects). Compared to a standard generalization bound which usually has a $\sqrt{1/n}$ term, our main technical innovation involves bounding the variance by the smoothness of the function class plus Poincaré-type inequalities. When $\beta$ in Theorem 1 is much smaller than $1$ we achieve a tighter bound. This is unique to the causal inference setting in which we are learning a function $f = \Psi(w,x) \to \tau$ among a family $\mathcal{F}$, where $w$ is intervention information (e.g. a drug’s attributes) and $x$ is individual features (e.g. patient medical history), and $f$ is assumed to be smooth with respect to $w$. **References** [1] Ruddigkeit, Lars, et al. "Enumeration of 166 billion organic small molecules in the chemical universe database GDB-17." Journal of chemical information and modeling 52.11 (2012): 2864-2875. [2] Curth, Alicia, and Mihaela van der Schaar. "Nonparametric estimation of heterogeneous treatment effects: From theory to learning algorithms." AISTATS. (2021). [3] Nie, Xinkun, and Stefan Wager. "Quasi-oracle estimation of heterogeneous treatment effects." Biometrika (2021). [4] Kennedy, Edward H. "Towards optimal doubly robust estimation of heterogeneous causal effects." arXiv (2020). [5] Curth, Alicia, et al. "Really doing great at estimating CATE? a critical look at ML benchmarking practices in treatment effect estimation." NeurIPS. (2021). [6] Yadlowsky, Steve, et al. "Evaluating treatment prioritization rules via rank-weighted average treatment effects." arXiv preprint arXiv:2111.07966 (2021). [7] Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." ICLR (2021). [8] Grover, Aditya, and Jure Leskovec. "node2vec: Scalable feature learning for networks." KDD. (2016).

Reviewer fcYE2023-08-14

Thanks for the clarifications. I agree formulation of zero-shot CATE estimation and appliction of ideas from other fields is not trivial, and have more than sufficient novelty for acceptance. Just make it clear in the paper. Overall, authors clarified some points, promised to revise some points and I don't see any major concern from fellow reviewers. So, I raise my evaluation by one point.

Reviewer o8uy2023-08-11

I have read the authors rebuttal and I will maintain my score of acceptance. I appreciate the detailed response on the subject of bias in \tau and the bias of the datasets. Looking forward to the updated paper with the supplementary experiments

Reviewer uxCL2023-08-18

Changing score

Updated my score based on the response and discussions

Program Chairsdecision2023-09-21

Decision

Accept (spotlight)

© 2026 NYSGPT2525 LLC