Detecting and Measuring Confounding Using Causal Mechanism Shifts

Summary

This paper introduces some measures of (conditional) confounding, based on information theoretic quantities. Some of their properties and an algorithm to estimate the measures using data from different environments. Although not included in the main text, the authors test their theory on some synthetic data.

Strengths

- The ideas on the paper are original as far as I know and I believe that the question regarding the measurement of confounding is an important one. The authors differentiate their work from previous work. - Beyond the errors outlined in the weakness section, the paper is well-written and presented. It is a bit unfortunate that the experiments were not included on the main paper.

Weaknesses

- There are several typos/mistakes on the paper: -- On line 158 and 169: $p$ is defined as the inequality between two distributions, this definition doesn’t make sense. I checked their reference 33 and 35 and it says in there that it is the probability of the distributions being different. Furthermore, p-value is a well-defined and known term, why overloading it? -- On definition 4.1 what they call directed information is not an -expected- KL divergence, it is simply a KL divergence. -- On Table 2, on rows 1 and 2 the equality with 0 is inverted. For example, if $X_i\to X_j$, then $P(X_i\mid X_j)=P(X_i\mid do(X_j))$ so that the log of such quantity is 0 for all values of $X_i$ and $X_j$, that is $I(X_i\to X_j)=0$. -- Is definition 4.6 really a definition? That sounds like a proposition to me. - I’m unsure of how large is the significance of the contribution being made (see questions and limitations). Which paired with the errors and typos above make me recommend for rejection of the paper.

Questions

I would like to get a feeling for why the properties of their measure of confoundedness are valuable (Theorems 4.2, 4.4 and 4.6). The only really essential property that I can think of such a measure is that it is 0 if the variables are non confounded and not 0 otherwise. Why for example do we want positivity? One could even be interested in a negative measure if the variables have a negative correlation under confoundedness, for example.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

This research does not contain any obvious negative societal impact. The authors do include a couple of sentences at the end of the paper where they state that there is a challenge to find real-world data to test their theory. Although I value the author's honesty about this limitation, I think it is too strong of a limitation.

Summary

The paper addresses the challenge of identifying and quantifying (unobserved) confounding in causal inference. They propose a more comprehensive approach by relaxing the classic assumption of causal sufficiency and leveraging the sparse causal mechanism shifts assumption. The authors introduce methods to detect and measure confounding effects, distinguish observed and unobserved confounding, and evaluate the relative strengths of confounding among variable sets. An empirical validation supports their theoretical analysis.

Strengths

(S1) The paper provides a thorough study of confounding from several perspectives: detecting and measuring confounding between pairs of variables and among multiple variables; distinguishing between observed and unobserved confounding; assessing the relative strengths of confounding among different sets of observed variables. To my knowledge, this is the first study exploring all these aspects in a unified framework. (S2) The paper is well written and easy to follow.

Weaknesses

(W1) I found the paper to be lacking in experimental evaluations. It is not clear how hard (both statistically and computationally) it is to compute the measures of confounding that were proposed, especially the ones measuring confounding among multiple variables. (W2) I am somehow skeptic about the practical relevance of the results presented in the paper (and the lack of experiments reinforces this point, see W1). In particular, people focused a lot on the marginal sensitivity model (and variations) as a measure of confounding strength because sensitivity analysis bounds can be easily derived from it. I am not sure the same would be possible under the proposed measure of confounding, and hence I am not sure how it would be useful in practice. (W3) I think some relevant related works are missing. In particular, when interventional data is available (e.g. RCTs) [1] and [2] propose testable implications for detecting hidden confounding. Further, [3] and [4] propose to lower bound the strength of hidden confounding (as measured by the marginal sensitivity model). [1] Falsification of Internal and External Validity in Observational Studies via Conditional Moment Restrictions. Hussain et al. AISTATS 2023. [2] Benchmarking Observational Studies with Experimental Data under Right-Censoring. Demirel et al. AISTATS 2024. [3] Hidden yet quantifiable: A lower bound for confounding strength using randomized trials. De Bartolomeis et al. AISTATS 2024. [4] Detecting critical treatment effect bias in small subgroups. De Bartolomeis et al. UAI 2024

Questions

(Q1) I got confused at lines 103-106: "While these methods use data from different contexts (an approach we also leverage in this work), they assume the absence of unobserved confounding variables; we instead focus on capturing both observed and unobserved confounding using data from multiple contexts.". [27] proposes a test for the presence of hidden confounding and hence allows for unobserved variables. Can the author briefly comment on how their measures compare with the approach proposed in [27]? (Q2) I am confused slightly confused with Assumption 3.2 (line 169): $$ P^c( X_i | \text{PA}_i) \neq P^{c'}(X_i | \text{PA}_i) $$ what does it mean for conditional distributions to be different? (Q3) Can the authors comment with one motivating example of how their proposed confounding measure could be used in practice? (See W2) [27] Rickard Karlsson and Jesse Krijthe. Detecting hidden confounding in observational data using multiple environments. Advances in Neural Information Processing Systems, 36, 2023.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

The authors mainly introduce capturing both observed and unobserved confounding using data from multiple contexts. Leveraging experimental data proposes a comprehensive approach for detecting and measuring confounding effects from three different settings that don't need the parametric assumptions and relaxes the causal sufficiency assumption. The authors provide the measure for detecting confounding effects (relative strengths of confounding). For each of the proposed measures, this article presents key properties and an algorithm for detecting and measuring confounding using data from multiple contexts.

Strengths

The entire article primarily introduces the detection and quantification of confounding effects without making parametric or causal sufficiency assumptions. It presents a well-structured logical framework to discuss this concept.

Weaknesses

I think overall the authors did interesting research, but my main concerns are listed below. 1. When the environment changes, for example, if $c$ changes to $c'$, will the original causal relationship change? 2. If the length of the environment node is only 1, will the method described in the article fail? 3. Why do settings 2 and 3 introduce mutual information to define confounding? Would there be any difficulties in using the KL divergence? 4. In line 104, the author said "an approach we also leverage in this work", but the author did not introduce the method in which article in detail. 5. In line 111, the article describes "we measure the effects of both observed and unobserved confounding". However, the subsequent sections do not appear to provide a direct method for measuring causal effects. 6. In fact, the confounding effect is not always a number between 0 and 1. Converting it to a number between 0 and 1 only measures its relative strength, but does not directly calculate its confounding effect. 7. In Setting 1: When there is confounding between two variables, CNF-1 is used to calculate it. If multiple variables have common confounding, CNF-1 is calculated for each variable with the others and then summed. Will this lead to the repeated accumulation of confounding? 8. The detect confounding method proposed in the article, but this measure is relatively rarely used in real life. In fact, it is unrealistic to only conduct simulation experiments. In this part of the experiments, the author did not specifically introduce the settings, so why were only the CNF-2 results output, and not the CNF-3 results? Is there a significant difference between these two results? For the section on "Measuring Conditional Confounding," the author should add experiments involving observed confounding, such as replacing the unobserved confounding with observed confounding in Experiment 1. 9. The authors repeat write “decision variables” in lines 149 or 150, and "two" in line 156. 10. The authors miswrite “$\mathcal{D^{C}}$” as “$\mathcal{D}$” in the third line of ALgorithm 1.

Questions

See the above Weaknesses.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors provide a detailed discussion of the limitations and applicability of their method.

Summary

This paper presents methods that: 1. define a measure of confounding between sets of variables, 2. separate the effects of observed and unobserved confounders, and 3. assess the relative strengths of confounding between sets of variables, in three different settings. In each setting they assume that data from several different contexts is available, and that the changes in causal mechanisms between the different contexts are known.

Strengths

This paper provides thorough theoretical justification for their methods. The analyses given for the latter two settings seem particularly novel to me.

Weaknesses

The assumption "that the causal mechanism changes are known for each variable across different contexts" seems to be quite strong, but is not mentioned until relatively late in the paper (line 160). The formal definition of a mechanism shift comes quite late in the paper. It would be nice to have intuitive explanations in the abstract and introduction. I also wonder if the formal notation could start with the notion of multiple environments baked in. As it is written now, it is unclear what the mechanism shift is relative to. Is it a shift between an environmental distribution and the interventional distribution, or is it only necessary to know the relative shift between environments? The relationship between definitions 4.2 and 4.3 and ignorability and conditional ignorability as independence and conditional independence assumptions needs to be added to the related works. There is also perhaps a connection to [Scalable Sensitivity and Uncertainty Analyses for Causal-Effect Estimates of Continuous-Valued Interventions](https://arxiv.org/pdf/2204.10022) that could be discussed. Notably, the relationship between the likelihood ratios and the KL divergence between the interventional and observational distribution mentioned in Appendix A.3.1. While the theoretical results are thorough, the empirical results are very limited. A major concern with hidden confounding is that it can arbitrarily bias estimated treatment effects. The magnitude of the bias should be straightforward to calculate in your synthetic experiments. It might be more demonstrative to show how the proposed measures correlate to induced biases. ## Points of confusion **Definition 3.1** - Looks like $X$ is being redefined here as to include any variable in $V$ rather than the subset that excludes $Z$. Mildly confusing. - line 131: $P(X_i \mid \mathrm{PA}_i)$ is not defined in definition 3.1 **Assumption 3.1** - talks about changes in causal mechanisms, but this concept hasn't been formalized yet which makes this assumption hard to understand. **Line 147:** what is a context node? what is an extended causal graph? Might help to define context specific distributions before rather than after. **Line 156:** two two **Line 156:** $P^c$ notation could be introduced earlier to formalize a mechanism shift. **Assumption 3.3:** $dPC$ is missing brackets **Line 182.** shifts -> shift **Line 183:** "extent of hidden confounding." word choice is a bit vague. maybe strength, or the effect of the confounder on each variable, or how much it may bias some estimand. **Definition 4.1:** The notation is a little awkward with the condition on $P(X_j)$ within $D_{\mathrm{KL}}$. **Theorem 4.2:** strongly -> more strongly **Definition 4.6:** missing definition of mutual information.

Questions

**line 128:** unclear notation, what information is $P_x(V), X \subseteq V$ adding here? If $P_*$ is a set, why not write the set that defines it instead? **line 128:** (i.e., $X = \emptyset$). why isn't $X$ bold? **Line 154:** "Let $C_{S \wedge \neg R}$ be the set of contexts in which we observe mechanism changes for the set of variables $X_S$ but not for the variables $X_R$." Changes relative to what? The interventional distribution? Just between two contexts? Is the shift the same across each of the contexts in this set? **Line 158:** This does not look like a p-value. This looks like an indicator taking a value of true or false, not a probability. Is there a missing $P$ outside of the brackets? If so, what is the relevant random variable? $c$? If $c$, confusing to have it lower-case while a potential value $X_i$ is upper case. Same questions for **Assumption 3.2**. **Theorem 4.1:** missing comma? $\\{X_iX_j\\}$ -> $\\{X_i, X_j\\}$ **Line 286:** Are the $\epsilon$ normally distributed? or is zero mean and any distribution ok? **Definition 4.7:** mutual information is defined between random variables. Are these random variables with respect to contexts? If so lower case $c$ is a bit confusing.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately discuss the limitations.

Answer to the rebuttal

I thank the authors for taking the time to answer my questions. About Table 2: you are absolutely right, I got confused with the causal graphs and the interventions when I was trying the measures out. About the real world examples and the properties of the definition: I appreciate those, and I would be ok if the paper was purely theoretical and had no experimental results, but then in that case I would expect very strong theoretical results, which in my opinion is not what the authors provide. In fact, the authors' answer to my question about the feeling of the properties seems to be that the properties are post-hoc; that is, they define some measure of confounding using information-theoretic quantities, they realise the properties hold because they defined it in such a way and then they write some Theorems based on that. In other words, I would say the authors have not answered my question of why the properties (besides monotonicity) are important for measuring confounding. I am happy to discuss this further, if the authors are interested in that.

A Note to All Reviewers and AC

We sincerely appreciate all reviewers for their time, insightful comments, and positive feedback. Reviewers' suggestions have significantly contributed to improving the presentation of our work. The majority of the suggestions from reviewers pertain to clarifications and editorial changes, which we addressed to the best of our knowledge (as also acknowledged by most reviewers). The reviewers' suggestions and comments are resolvable and feasible to be included within the allowed 10 pages (we will use the additional page granted if the paper gets accepted). We have prepared an updated manuscript incorporating the following changes: 1. We have moved the experiments from the appendix to the main paper and added the new experiments shown in the rebuttal PDF to the main paper. 2. The two real-world examples presented in the rebuttal PDF are now included in the introduction. 3. The relationship to sensitivity analysis, as suggested by reviewers K6Rd and n1WD, has been added to the related work and methodology sections of the revised manuscript. 4. All typos and other editorial comments have been addressed in the final manuscript.