Proximal Causal Inference With Text Data

Summary

The situation with unobserved confounding variables is quite common in applied research and makes causal inference complicated. To deal with such settings, the authors propose a new causal inference method that uses multiple instances of pre-treatment text data, estimates two proxies from two zero-shot models on the separate instances, and applies these proxies in the proximal g-formula. It is shown that the identification conditions are satisfied, The method is evaluated in a simulation study on synthetic and semi-synthetic data.

Strengths

- The paper proposed a new innovative method for proximal causal inference utilizing text data. - The paper is well written and motivated.

Weaknesses

- Literature review is incomplete. The idea of proximal inference goes back to Zvi Griliches (1977) who should get credit for it. As background reading the corresponding chapter in the textbook "Applied Causal Inference Powered by ML and AI" by Chernozhukov and coauthors (2024). - To overcome the completeness condition the paper "Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approach" (arXiv:2103.14029) might be helpful. - The theoretical results (Prop 1-4) are all in some sense "negative" and straightforward / already known. What would be more interesting, would be some "positive" results about the proposed method, although I am aware that this might be quite challenging.

Questions

- How credible / realistic are (P1) and that $T^1_pre$ is conditionally independent from $T^2_pre$ in typical applications (e.g. using electronic health records)? - How could one use EHR to construct proxies? Are the assumptions fulfilled? (Might be questionable.. the example mentioned in the text is not convincing.)

Rating

5

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

Limitations are adressed adequately. As mentioned above, theoretical guarantees for the propsoed method might be very valuable.

Summary

Causal inference techniques often rely on the assumption that all confounders can be observed or inferred from available data. This paper proposes a method to address the setting where a confounder is entirely unobserved; instead there is available pre-treatment text that can be used to infer proxies, which in turn have predictive value for the unobserved confounder. The authors adapt proximal causal inference ideas to text-data settings, discuss "gotchas" in naive implementations of such a method, and explore various relevant choices before presenting best performing method. Specifically, they use multiple instances of pre-treatment text to infer two proxies for each datapoint using two different zero-shot language models. This is then used for average causal effect estimation via the proximal g-formula. The paper provides empirical evaluation on synthetic and semi-synthetic datasets as well as a falsifcation heuristic for untestable assumptions.

Strengths

Originality: While proximal causal inference has been proposed and studied, this paper adapts existing methods to text data and explores practical considerations in their implementation. Clarity: The paper is well-written, with clear statements of the requirements of the method and the assumptions it relies on. Quality: The paper proves various propositions that guide the design of the proposed method in a principled manner. It also explores a falsification heuristic for untestable assumptions and empirically verifies that the heuristic is aligned with final performance in synthetic and semi-synthetic settings. Significance: Unobserved confounding is a significant challenge in causal inference. The paper is well-motivated by the availability of large amounts of text data and the recent progress in language modeling.

Weaknesses

- The significance of the empirical evaluation is limited due to synthetic and semi-synthetic experiments. The impact of the proposed method in realistic settings is hard to assess. While the evaluation of real settings is hampered by the lack of ground-truth, it may still be possible to synthesize data that is as realistic as possible so that the more realistic challenges of deploying such a method may be uncovered. - The experiment design of the paper carefully avoids post-treatment text to avoid biased effect estimation. However, the proxy inference from language models indirectly depends on the pretraining dataset used to train the language model. The data in that pretraining set may be post-treatment text. The impact of this dependence is unclear. - It is unclear how sensitive the performance of the proposed method is to the choice of pretrained zero-shot model, or whether we can expect better performance as the zero-shot model used scales up.

Questions

As mentioned above: - Do the authors expect the performance of their method to improve as they use larger language models to infer proxies? - Can the authors provide any intuition for what they expect to be the impact of using a language model that has been trained on post-treatment text data to infer proxies and plug them into the causal effect estimation?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper sufficiently addresses limitations of the setting and method proposed.

Summary

The paper considers the causal effect estimation setting where a confounder is latent but with unstructured text data that could serve as proxies. Specifically, the paper proposes to incorporate zero-shot classifiers (to operate on text-based proxies), together with a falsification heuristic, into the proximal causal inference framework. The goal of the text-based proxy design is to craft W and Z that satisfy the identification condition in proximal causal inference framework. Empirical evaluations are presented in synthetic and semi-synthetic experiments. --- **Post rebuttal** I have increased my score, under the assumption that the revised manuscript could sufficiently address the original concerns on material organizations.

Strengths

The strength of the paper comes from the attempt to conduct proximal causal inference on unstructured text data. The paper carefully considers identification conditions specified in the proximal causal inference framework, and proposes a design procedure together with a falsification heuristic to find two text-based proxies (so that the proximal inference can be conducted).

Weaknesses

The weakness of the paper comes from the organization of the material (especially the constraints/conditions involved), and relatively simple settings considered in (fully-/semi-) synthetic experiments. In particular, further clarifications and/or discussions on following points would be very helpful (detailed in "Questions" section): (1) regarding a series of different constraints, conditions, gotcha's, assumptions (2) the fully and semi- synthetic experiments consider structured data, it is not exactly clear how these text proxies look like and if they correspond to all or part of the aforementioned constraints/conditions. Examples of text proxies would be much more intuitive than claiming "satisfy by design" (line 136)

Questions

(1) regarding a series of different constraints, conditions, gotcha's, assumptions In Section 2, there are P1 -- P4, a set of conditions should be satisfied in order for the proximal g-formula to work. In order to respond to the criticism of potential unavailability of W and Z in structured data, further assumptions S1 -- S2 are proposed. Then in Section 3 and Section 4, a set of gotcha's, two additional pre-conditions (lines 216 -- 218), and another set of conditions related to odds-ratio based heuristics (lines 245 -- 247) are presented. How do these conditions fit together to respond to the criticism of proximal causal inference on structured data? Are they all identification conditions for the text-based proxy proximal causal inference (other than the fact that they are conditions specified by previous works for different components put together)? (2) the fully and semi- synthetic experiments consider structured data, it is not exactly clear how these text proxies look like and if they correspond to all or part of the aforementioned constraints/conditions For fully and semi- synthetic experiments, the setting is largely structured data instead of text-based proxies. Examples of text proxies that the proposed design procedure yields would be much more intuitive than just claiming "satisfy by design" (line 136).

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The organization of material could be improved to better present how the proposed approach responds to criticism of proximal causal inference (with structured data). Examples beyond fully and semi- synthetic experiments will make it clearer w.r.t. how the designed text-based proxies help to address the aforementioned criticism on previous work.

Comment

First of all, thanks a lot for your efforts and the detailed answer. I still think that the negative results in Proposition 1-3 are not really deep and well known. I am aware that the reference can be considered as a complement, but I think a thorough literature review should contain all related sources. Finally, I still find the assumption of conditional independence hard to hold and I think here more convincing examples would be helpful. Overall, I will keep the score.