Covariate Shift Corrected Conditional Randomization Test

Summary

The paper introduces a novel approach to addressing covariate shift in conditional independence tests. By leveraging importance weights and the control variates method,the paper proposes Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test, which maintains asymptotic Type-I error control. In addition, a power enhancement is proposed, which reduces the variance in the test statistic and thus improve the power towards alternatives. The methodology is validated both theoretically and empirically, demonstrating superior performance in simulation studies and practical application to a COVID-19 treatment dataset, highlighting its efficacy in real-world scenarios.

Strengths

Originality The paper presents original contributions by introducing a new approach to conditional independence testing under covariate shift. This method combines the idea of conditional randomization test, importance weighting, the control variates technique. Quality The work is of high quality, demonstrated through rigorous statistical theories on the validity and power, and comprehensive empirical validations. Clarity The paper is clearly written and well-organized. Each section has a clear message and readers can quickly grasp the main idea. Significance I consider this work as a significant contribution to the field of conditional independence testing and covariate shift. The method is easy-to-implement, and is more powerful than existing methods as shown in numerical experiments.

Weaknesses

The paper does not leave serious technical weaknesses, but I am concerned about the limited applicability of this method in real-world problems. Based on my understanding, one would test for conditional independence directly on the target distribution if the outcomes in the target population were easy to collect. So csPCR is useful only when the outcomes are costly to collect. This questions several application examples mentioned in the paper. First, in the college admission example (from Line 37-53), why don't the economists just collect the college admission results from the target population and then conduct the vanilla PCR? Typically, the target outcomes have to be collected for followup analysis after the conditional independence test. Second, Section 5 introduces a real-world application of this method on Covid-19 pandemic data, where the source and the target data set are segmented based on time. Given such a sequential data collecting process, it is unclear whether the iid assumption still holds in the source and target distribution. See Questions for more details. Despite of the concerns above, I do believe that this method is useful for certain genetic and biomedical problems, where the outcomes are difficult to collect, but other covariates (such as SNPs) are relatively easy. Therefore, I suggest the authors clarify the application examples a bit by discussing more reasonable applications with some references.

Questions

1. Page 5, Algorithm 1: Besides the variance issues considered in Algorithm 2, it is clear that the choice of test statistic in Eq (4) also affects the power. Can you comment on how to choose the test statistic? In the numerical experiment, the test statistic is specified to be the product of $Y$ and $X$, which makes sense as it measures the "correlation" between Y and X. However, $Z$ is not included in the test statistic, which seems to decrease its power. Have you tried any other test statistics? Do difference choices affect the power significantly? For instance, one alternative is to use $(Y - \beta_y Z)(X - \beta_x Z)$, where $\beta_y and \beta_x$ are regression coefficients for $Y\sim Z$ and $X\sim Z$ on an independent data set from source distribution. 2. Page 4, Line 170-177: The paper argues that they used PCR as the baseline test because PCR is more powerful than the vanilla CRT. But on the other hand, CRT is also better than PCR in the sense that it has finite-sample type I error control, rather than asymptotic type I error control. It's worth mentioning this point as well. While I notice that the simulations in the paper shows satisfactory type I error control, the type I errors may get inflated if, for example, the data have heavy tails. 3. Page 9, Section 9: Relating to the weaknesses, have you run any tests to check the iid assumption on the Covid-19 data set?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The papers discuss about the limitations of their method under model misspecifications in Section 4, pointing out that the powerful enhancement can disappear when there is a full nonlinear component in the model. The authors have addressed the limitations adequately.

Summary

This paper proposes a new variation of CRT to be applied in the presence of covariate shifts. The paper presents a method and one extension for each, with higher power. Then, the authors present some needed theoretical results and finish with experiments.

Strengths

- The paper proposes a theoretically correct approach for testing CI under covariate shift; - The paper presents their algorithms and theoretical results; - The paper had convincing experiments;

Weaknesses

- It would be interesting if the authors could estimate the full density ratio in their simulations (possibly in a high-d scenario) and then compare their results with the IS approach; - The paper does not conduct any experiment where Type-I error control is shown in a real dataset, e.g., like for example in [1,2]. References [1] Pogodin, R., Schrab, A., Li, Y., Sutherland, D. J., & Gretton, A. (2024). Practical Kernel Tests of Conditional Independence. arXiv preprint arXiv:2402.13196. [2] Maia Polo, Felipe, Yuekai Sun, and Moulinath Banerjee. "Conditional independence testing under misspecified inductive biases." Advances in Neural Information Processing Systems 36 (2023): 58577-58612.

Questions

- How can we understand the role of the effective sample sizes (weight imbalance) in the effectiveness of your method? - Could you elaborate more on the surrogate variables? I am familiar with the literature on CI testing, but I haven't seen the presence of these variables in other works.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors comment on limitations.

Summary

This paper introduces the Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test, designed to address covariate shift in conditional independence testing. The csPCR method incorporates importance weights and employs the control variates method to enhance test power and reduce variance in the statistical analysis. Theoretical contributions demonstrate that csPCR controls Type-I error asymptotically. Empirical validations through simulation studies and a real-world application assessing COVID-19 treatment effectiveness showcase the method's effectiveness.

Strengths

1. **Theoretical contribution:** The paper makes a theoretical contribution by addressing the covariate shift in conditional independence testing. 2. **Methodological Innovation:** Introduction of the Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test that incorporates importance weights and control variates method to manage covariate shifts effectively. 3. **Empirical Validation:** Extensive simulation studies and real-world application (assessment of COVID-19 treatment on 90-day mortality) demonstrate the practical efficacy and superior power of the proposed csPCR test over traditional methods.

Weaknesses

1. **Dependence on Accurate Estimations of Density Ratio:** The csPCR test's performance is critically dependent on the accurate estimation of density ratios. This dependence could pose significant challenges in practical scenarios characterized by limited, noisy, or high-dimensional data, where reliable density ratio estimation becomes inherently difficult. 2. **Clarity on the Advantage Over Resampling Methods:** The paper does not explicitly clarify the advantages of the csPCR test over simpler resampling-based methods that also utilize estimated density ratios. While the csPCR integrates density ratios directly into the test statistic calculation and employs variance reduction techniques, it remains crucial for the authors to demonstrate why these features offer substantial improvements over traditional methods that adjust the sample weights during resampling. The discussion should address whether the complexity of csPCR provides tangible benefits, such as improved error rates or robustness in more varied practical applications, compared to potentially simpler resampling approaches.

Questions

- Could the authors provide a detailed comparison between the csPCR test and traditional resampling methods that also use estimated density ratios? This would further strengthen the paper - The paper mentions the use of control variates to reduce variance introduced by importance weights. Could the authors discuss how this approach compares to variance reduction techniques used in resampling methods? - Could the authors elaborate on any theoretical limitations or potential failure modes of the csPCR method? - Are there additional empirical validations planned or underway to test the csPCR method across more varied datasets?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors are encouraged to add a separate limitation section in either the main paper or appendix.

Summary

This paper addresses the issue of conditional independence under covariate shift. The authors' goal is to test the conditional independence for causal inference in the target data. The authors can use the source data whose distribution is potentially different from that of the target data. For this problem, the authors propose using the covariate-shift adaptation method and develop a method for testing the null hypothesis.

Strengths

This paper addresses the issue of conditional independence under covariate shift, which I find to be an intriguing problem. However, I was unable to comprehend the underlying assumptions of this problem, making it difficult to proceed with the reading. I understand the objective of this manuscript to be as follows: 1. The authors aim to test the null hypothesis $H_0: X \perp Y \mid Z$ for the target data (Here, I denote the independence by $\perp$). 2. There is a possibility that $X \perp Y \mid Z$ does not hold simultaneously for both the source and target data. 3. To test the null hypothesis for the target data, the authors employ an algorithm adapted to covariate shift. My primary concern is whether the assumption "there is a possibility that $X \perp Y \mid Z$ does not hold simultaneously for both the source and target data" is justified in the first place. The authors assume that, in order to use the covariate shift adaptation algorithm, - $Y \mid X, Z, V$ is the same in both the source and target data. Under this assumption, is it not the case that the situation where $X \perp Y \mid Z$ holds in one dataset and does not hold in the other cannot exist? $X \perp Y \mid Z$ implies that $p(y, x \mid z) = p(y \mid x, z) p(x \mid z) = p(y \mid z) p(x \mid z)$, meaning that $p(y \mid x, z) = p(y \mid z)$. Given that we assume $Y \mid X, Z, V$ to be the same in both the source and target data, does this not imply that whether $X \perp Y \mid Z$ holds cannot differ between the source and target data? Due to this ambiguity, I was unable to further evaluate the paper. I would like the authors to clarify this point so that I can proceed with the evaluation.

Weaknesses

See above.

Questions

See above.

Rating

4

Confidence

1

Soundness

2

Presentation

2

Contribution

2

Limitations

See above.