Summary
The paper under review offers a thoughtful exploration of how model misspecification influences conditional independence testing. The significance and practical applications of such testing are aptly underscored in the introduction. The use of a toy example to demonstrate the impact of model misspecification on Type I error is a compelling approach that effectively propels their study. However, the paper's structure could be improved for better readability. While the work is technically exhaustive and appears sound, the layout requires refinement. For instance, the discussion and conclusion section seems truncated due to an overflow of technical details, presumably in an attempt to conform to the page limit.
Additionally, the paper presents an insightful study on Type I error bounds and illuminates how factors in model misspecification might influence the performance of our model. This is examined across three leading methods of regression-based conditional independence testing, STFR, GCM, and RESIT; a commendable breadth of investigation. On the other hand, there lies a lot of ingenuity in the application of sufficient statistics and the Rao-Blackwell theorem to devise a conditional independence test - an approach that stands out as particularly clever.
Despite the paper's notable theoretical contribution and illumination of model misspecification implications, its current state warrants a rejection recommendation. The primary rationale lies in the sparse experimental validation, which leaves room for further expansion. While a more comprehensive set of experimental results could potentially sway my assessment towards endorsement, the manuscript, as it stands, along with the limited set of experiments, doesn't quite meet the high standards necessary for acceptance. Thus, my score, on balance, falls on the lower side.
Strengths
1. The paper exhibits a robust theoretical foundation, ingeniously leveraging the principles of sufficient statistics and the Rao-Blackwell theorem to address conditional independence testing. This innovative application, to the best of my understanding, introduces a novel concept and forms a significant contribution.
2. The paper provides extensive misspecification bounds that comprehensively span across various regression-based methods. Moreover, the experiments also benchmark against simulation-based approaches.
3. The exploration of examples in Appendix A.2 is highly instructive, where model misspecification can influence and Type I error of our test significantly. In addition, the choice of synthetic experiments adapted from Berrett et al. serves as a fitting means of demonstrating the concept in this study.
4. In an innovative deviation from the traditional setting, this paper presents proofs across a series of models that converge to a pre-established class of models, thereby aligning well with contemporary machine learning paradigms
Weaknesses
1. Acknowledging that the paper's paramount contribution resides in its theoretical development, I must express my reservations regarding the experimental element, which appears somewhat insufficient. To bolster the validation process, I've proposed additional experimental situations in the "Questions" section. As it stands, the validation primarily leans on a single toy experimental setup, and a comprehensive benchmarking across different methods in a real-world setup is lacking.
2. In terms of presentation, the paper could benefit from some refinement. Notably, a significant portion of the proof concepts across different Conditional Independence Testing methods appears to overlap. A majority of these proofs could have been concisely presented by adopting lemmas that generalize across multiple methods, thus increasing reader engagement. Also, the main body is currently dense with scattered technical terms that could be better consolidated.
3. Just a minor point of clarification, I'd like to delve into the details of the conducted toy experiments. Developing a unified framework that frames functions as a sequential path towards a distinct model group is truly commendable. Nonetheless, there's scope for refining its presentation. Specifically, the introduction gave the impression that this was a broad generalization of model misspecification, which it doesn't appear to be.
Upon incorporating regularization terms during training, it's important to note that while the overarching model class remains constant, the subset of models that surface must adhere to the regularization-imposed limits. Consequently, the model's prime focus may deviate from merely minimizing the Bayes predictor loss.
This deviation could result from either integrating regularization terms into the optimization formula, thus modifying the loss characteristics, or from imposing these terms as constraints, effectively limiting the model class.
Considering these factors, I'd suggest positioning your model misspecifications as a tool to tailor the concept to real-world machine learning environments, rather than representing it as a broad generalization of the concept of model misspecification itself.
4. One of the method's most significant practical implications, in my view, is the potential for semi-supervised learning on unlabeled data - a scenario where RBPT is likely to outshine RBPT2. Given its importance in practical real-world settings, this deserves an additional set of experiments. An easy experiment to consider would be to add a large set of unlabeled data to the toy experiment.
Questions
1. Regarding eq. (3.1), it seems to me that $\hat{\sigma}$ mismatches the sample standard deviation of $T_i^{(m)}$; it is incorrectly expressed as a biased estimator. For simplicity, you might want to just denote $\bar{T}^{(n,m)}$ and $\hat{\sigma}^{(n, m)}$ as the sample mean and variance estimators and not include any formulas, referencing Dei et al. for looking up the exact derivations.
2. In the experiments section, especially concerning figure 2, I have several points of interest:
* It would be insightful to observe how RBPT behaves in terms of Type I error when $Q$ is misspecified. For instance, a scenario where $c=0$, $\theta=1$, and $\gamma$ is progressively increased could be worth examining. Alternatively, to evaluate how sensitive RBPT is to such misspecification, setting $\gamma$ to a larger value (say 0.02), and sweeping through different $\theta$ values to assess Type I error might be valuable. I believe this experiment is of extreme importance since, in most real-world situations, the total variation between $Q^*(X|Z)$ and $P(X|Z)$ is significant and may not be bounded by the Jensen gap. Assessing this sensitivity is pivotal to demonstrating the robustness of the setting. The second plot, where $\gamma = 0$, may not adequately reflect this robustness.
* It would be beneficial to include the values of other hyperparameters set in the caption of Figure 2. This would prevent the need to navigate back and forth through the text to understand the setup; also, I assumed that for the third plot $\gamma = \theta = 0$, but it was not mentioned in the main text.
3. Why hasn't the power of RBPT been directly compared with the baselines for the real-world examples? The reasons for not contrasting RBPT are somewhat elusive to me. While I concur that both RBPT and RBPT2 theoretically establish bounds for Type I error control under model misspecification, a quantitative comparison with baselines is still essential due to the trade-off between Type I and Type II errors. Although we don't expect RBPT to outperform other models in terms of power, it would be interesting to see if it possesses comparable power, as suggested by the third plot in Figure 2.
4. I am perplexed by the stark disparity between the performance of RBPT2 in low versus high-dimensional settings. It appears that an increase in the dimension of Z invariably leads to rejection of the null hypothesis, resulting in a rise in both power and Type I error rate. Could this be attributed to the need for a dimension normalization term in $b$? How are $b$ and $a$ generated per experiment, and how does $b^T Z$ scale with the dimension? Why doesn't this impact RBPT or CPT? After all, one of the main applications of regression-based conditional independence tests are high-dimensional settings, and it is important to address such concerns.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
1. In the RBPT methodology, model misspecification is conceptualized as the difference between $Q^*(X|Z)$ and $P(X|Z)$, and between $g^*(x, y)$ and $f^*(x, y)$. This deviates from prior methods where the misspecification is solely defined between $g_1^*$ and $f_1^*$, and $g_2^*$ and $f_2^*$. While the latter defines misspecification between functions, the former is defined on an entire distribution.
Notably, even if we posit that the disparity between the predicted $\hat{Q}(X|Z)$ and $Q^*(X|Z)$ is infinitesimal, the divergence between $Q^*(X|Z)$ and $P(X|Z)$ remains far from negligible. Furthermore, the effectiveness of the entire method hinges on this misspecification gap ($\Omega_{P,1}^{\text{RBPT}}$) being smaller than the Jensen's gap, which places stringent restrictions on its applicability. While theoretically, this method relaxes the bound on the expected total variation between $Q^*$ and $P$ and sets it to $\frac{\Omega_{P,2}^{\text{RBPT}}}{M \cdot L}$, the $M$ and $L$ coefficients will be pretty large in practice. On the other hand, while RBPT2 appears to circumvent this issue, its theoretical grounding isn't as robust as that of RBPT, thereby reducing its perceived solidity.
I believe this issue will become more prominent in real-world settings and the toy experiments do not reveal this core flaw of the method. Therefore, the authors should have mentioned this limitation or run extra experiments to show the effect of this gap.
2. A prominent constraint associated with RBPT2 appears to be its performance when scaling $Z$. Unfortunately, this aspect is not addressed within the main paper text. It would be beneficial to incorporate some commentary on this issue, as it pertains to the most plausible plug-and-play application. Furthermore, I harbor concerns regarding the potential for error from the initial regression to percolate into the subsequent one, which could potentially escalate the overall margin of error. This is pointed out in Appendix A.6, and with the empirical results acting strangely with scaling (I also mentioned in the “Questions” section), I am concerned as to how much this can hinder the performance of RBPT2.