Cause-Effect Inference in Location-Scale Noise Models: Maximum Likelihood vs. Independence Testing
A fundamental problem of causal discovery is cause-effect inference, learning the correct causal direction between two random variables. Significant progress has been made through modelling the effect as a function of its cause and a noise term, which allows us to leverage assumptions about the generating function class. The recently introduced heteroscedastic location-scale noise functional models (LSNMs) combine expressive power with identifiability guarantees. LSNM model selection based on maximizing likelihood achieves state-of-the-art accuracy, when the noise distributions are correctly specified. However, through an extensive empirical evaluation, we demonstrate that the accuracy deteriorates sharply when the form of the noise distribution is misspecified by the user. Our analysis shows that the failure occurs mainly when the conditional variance in the anti-causal direction is smaller than that in the causal direction. As an alternative, we find that causal model selection through residual independence testing is much more robust to noise misspecification and misleading conditional variance.
Paper
Similar papers
Peer review
Summary
The paper discusses the sensitivity of the recently introduced heteroscedastic noise model to model specification for causal discovery in a bivariate scenario. Specifically, it focuses on the impact of misspecifying the assumed noise distribution, which leads to poor performance when choosing the direction based on maximizing the likelihood. Instead, the paper proposes the use of independence testing of the residuals and presents an improved model. The newly proposed model demonstrates promising results in both real-world and synthetic datasets.
Strengths
* Excellent and clear introduction. * Comprehensive literature review with useful categorization of different approaches. * Highlights flaws in the existing prior approach. * Extensive experiments using popular causal discovery datasets. Also see questions section.
Weaknesses
* The novelty appears to be limited: The result seems rather incremental, considering that the implications of model misspecification aren't surprising, and the use of independence tests is a relatively common way to utilize functional causal models for causal discovery. * There is a limited discussion about the work "Identifying Patient-Specific Root Causes with the Heteroscedastic Noise Model" by Strobl et al., given that they don't rely on maximizing the likelihood for fitting the model, do not seem to depend on prior knowledge about the noise distribution and also focus on causal discovery. Also see questions section.
Questions
The paper is generally well-written and includes an in-depth comparison with related work. Some general remarks and questions: * You mention LiNGAM as an independence testing approach. However, strictly speaking, it only relies on the ICA components (assuming non-Gaussian noise), which is not utilizing independence tests. * The reference for GRCI appears to be incorrect. * There needs to be a more comprehensive comparison with the work by Strobl et al., as they do not rely on prior knowledge about the noise or exploitation of the conditional variance, which is the main critique of this paper. * The meaning of Figure 1 could be clearer. For instance, what does “high” and “low” exactly mean for accuracy? * When considering the conditional variance, it seems evident that this is primarily a matter of scaling. One can simply scale X and Y differently to arbitrarily flip edges for approaches that do not carefully scale the variables as part of their methodology. * Table 1 seems slightly out of place and appears to be more appropriate for the experiment section. * In the case of CAREFL-M, isn't the prior distribution merely a convention, since they learn a transformation function of the noise, i.e., they use g(N) and not N directly. Therefore, the Gaussian noise can be transformed into 'arbitrary' noise distributions. Perhaps, I'm missing something here? --Update after rebuttal-- I have read the rebuttal and further discussed with the authors. Many of my concerns have been addressed and I raised the score by 1.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
No concerns regarding societal impact. Regarding technical limitations, this paper focuses primarily on improving an existing methodology and, thus, shares the same limitations as the original work. However, it aims to make the methodology less sensitive to model misspecifications.
Summary
This paper explores causal discovery using heteroscedastic location-scale noise functional models (LSNMs) (which generalizes the additive noise models) and compares maximum likelihood model selection to causal model selection through residual independence testing.
Strengths
1. I think this paper is significant in that it provides empirical and theoretical evidence of using the IT method for causal discovery. 2. This paper is well-written. The contribution is very clearly written from the introduction, so little chances to miss the contribution of the paper. 3. The simulations are extensive.
Weaknesses
1. I think the robustness of the IT method should be more theoretically advocated. 2. Some results (e.g., Proposition 5.1.) don’t have proofs, and I am not sure they are trivial.
Questions
- Q1. What is the implication and usage of Lemma 4.1 in the paper? - Q2. What is the definition of noise misspecification and misleading CV? Does it mean that the estimation model for the residual (noise) is wrongly specified? - Q3. “(Identifiability) When the form of the noise distribution is correctly specified, the causal model always has a higher likelihood (Figure 2a).” can be proved by Theorem 4.2 or Corollary 4.3? If so, could you please provide easy and intuitive explanations? - Q4. I think the proof for Proposition 5.1 should be provided.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
I believe that more theoretical analysis is necessary to claim the robustness of IT. For instance, does Proposition 5.1 imply that the IT-based method is robust against noise misspecification and misleading cross-validation? If we assume regression suitability in the machine learning method, will the result change for ML?
Summary
The authors investigate bi-variate causal inference with non iid, heteroscedastic noise. They specifically investigate a group of causal inference methods based on max-likelihood and present two main drawbacks: max-likelihood approaches with parametric assumptions on the noise distribution, such as normality, suffer when these distributions are misspecified. Further, the authors show that max-likelihood is inversely proportional to conditional variance. Hence, if the conditional variances are misleading the max-likelihood points in the anti-causal direction, which is the case for many pairs in the Tübingen real world cause-effect benchmark. The authors propose an independence test based approach fitting an affine flow to model the conditional density and HSIC test the residuals.
Strengths
- Investigating the assumptions and failure modes of bi-variate causal inference methods is an important to ensure that these methods are accurate under real-world conditions. - The authors provide a thorough argument against max-likelihood based methods based on theory and empirical evidence. The empirical experiments demonstrate how misspecified noise and noise level affect the accuracy of max-likelihood methods. - The linkage of max-likelihoods failure modes to misleading conditional variance is plausible and is backed by theory. - The proposed method CAREFL-H performs well in the benchmarks covering synthetic and real world data. It outperforms both independence test and maximum likelihood based approaches alike.
Weaknesses
- The contribution of this work is limited in light of recent work. It partially overlap with the work on noise misspecification by Strobl & Lasko 2022, and the critic on likelihood-based causal inference by Schultheiss & Bühlmann, 2023. - The pitfalls of independence test methods is not discussed, leading to an incomplete picture in regards to comparing both families of approaches. - The proposed amendment to the CAREFL method introduces only an independence test afterwards and applies the same scheme (RESIT, GRCI) with different function modelling tools. - The claim, that affine flows are "empirically suitable under noise misspecification and misleading CVs" is not sufficiently supported by the experiments which cover only limited function classes (sine, sigmoids). - For instance, CAREFL-H performs worse than RESIT on SIM, which also uses HSIC with a different function estimator that works than the affine flows. - Finally, the presentation including writing and figures is unclear at times (for details see minor remarks). References Schultheiss, C., & Bühlmann, P. (2023). On the pitfalls of Gaussian likelihood scoring for causal discovery. Journal of Causal Inference, 11(1), 20220068. Strobl, E. V., & Lasko, T. A. (2022). Identifying patient-specific root causes with the heteroscedastic noise model. arXiv preprint arXiv:2205.13085. Minor Remarks: - Figure 2 plots are squished. - Table 2a) is very cluttered and lacks a caption, maybe better as line chart to see trend. - Theorem 4.2 is a main result of the paper but not explained sufficiently, i.e. only half a line long and the following paragraph is also very sparse. - The acronyms ML and IT are associated with machine learning and information technology and hence easy to confuse, consider using different acronyms.
Questions
- Table 1: If conditional variance is misspecified, i.e. higher in the causal direction, how do the M methods still get 100% right for all gaussian high noise pairs? And why does it not work for uniform noise? - What could be some benefits of using max likelihood, for example, if learned function \hat{f} or scaling \hat{g} is misspecified, my intuition is that max likelihood has more slack than independence tests. - What is the sample complexity of CAREFL-H, how much is it increased as both a train/test set is needed, where the test set needs to be large enough for an HSIC test with sufficient statistical power. - Experiments: you test noise independence after a regression with CAREFL-H same as RESIT, why is RESIT better on SIM across the board?
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
The downsides of independence testing approaches are not addressed. Hence, the comparison between independence testing and max-likelihood methods is not complete. For example, in the homoscedastic case max-likelihood methods (CAM) outperform independence test methods (RESIT). This hints towards max-likelihood methods also having an edge against independence tests in some cases.
Summary
This paper compares using maximum likelihood vs using tests of independence with regression residuals to select between a causal model and an anti-causal model, in order to infer the causal direction between two random variables. An empirical observation is that assuming a location-scale noise model, the maximum likelihood approach tends to perform poorly when the noise distribution is misspecified and the conditional variance in the causal direction is greater than that in the anti-causal direction. By contrast, the independence testing approach or at least the particular affine flow model used in this paper appears to maintain good performance even in such unfavourable circumstances. In addition to extensive empirical results to establish this phenomenon, this paper also provides some theoretical explanations of the phenomenon.
Strengths
This paper is very well written and readable. The empirical results are convincing and illuminating. The theoretical results are interesting and provide a compelling and useful explanation of the empirical observation. Given the importance and difficulty of the problem of inferring causal asymmetry from observational data, the empirical results and theoretical analysis presented in this paper strike me as sufficiently significant.
Weaknesses
As far as I am concerned, this paper does not have notable weaknesses. I would appreciate some explanation or even speculation on why the flow estimator remains empirically suitable under noise misspecification and misleading conditional variances. Does noise specification play any role in the flow estimator?
Questions
Does the specification of noise distribution play any role in the flow-based independence tests? If not, I wonder if that is the main reason why it is more robust than the maximum likelihood methods. How much, if at all, do misleading CVs by themselves (i.e., without noise misspecification) affect the performance of the ML methods?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
OK.
I want to thank the authors for their thoughtful response to my raised concerns. > Re: Weakness, Little Discussion of GRCI I may be missing a detail, but the GRCI method does not appear to require any assumptions regarding the noise distribution since they do not depend on the likelihood maximization approach. Hence, investigating the impact of a misspecification seems only relevant if the LSNM model is learned using an approach, such as in [9], that requires explicit assumptions about the noise distribution. Seeing this, it appears one does not need to worry about noise misspecifications if the GRCI approach is used. > Re: Questions, Scaling You mentioned that "in ANMs, scaling may lead to a flip in likelihoods". However, one would typically use ANMs for testing the independence between the residual and input (e.g., after fitting a model in a least-squares manner). Such a test should remain scale-invariant. Even if we look at the likelihood of ANMs, if the ANM is correctly specified (like you assume for the noise of LSNMs), shouldn't this also prevent flipped edges? > Re: Questions, Isn't Gaussian Distribution Merely a Convention Thank you for the clarification. I understand that g(X) is a scaling factor in the LSNM, but I was referring to the CAREFL model, which is characterized as X = f(X_pa, g(N)), where g is an invertible function. I apologize for the confusion due to overlapping notation. In CAREFL, there's no need to specify the correct noise distribution, since it aims to learn a model up to a (non-linear) invertible transformation of the noise which is sufficient. This is discussed, for instance, in Section 5 of "Counterfactual (Non-)identifiability of Learned SCMs" by Nasr-Esfahany et al. That being said, I understand that these points are generally less related to your work since you are rather investigating the impact of misspecifications in case of a likelihood approach to infer LSNMs.
Thank you very much for the follow-up and clarification! > Re: Questions, Isn't Gaussian Distribution Merely a Convention CAREFL [11] employs $N$ not $g(N)$ as the model prior. Please refer to the following excerpts from [11]: - $u$ in Equations 5 and $z_j$ in Equation 6 - "The prior of each flow is fixed to a Laplace distribution" - Figure 2 - "All flows use an isotropic Laplace distribution as a prior." - Appendix F.1 Architectures and hyperparameters Therefore, CAREFL is in the form $Y=f(X) + g(X) \cdot N$, not $Y=f(X) + g_{1}(X) \cdot g_{2}(N)$ or $Y=f(X, g(N))$. So, learning an invertible transformation of the noise does not apply to CAREFL. Furthermore, if CAREFL were in the latter two forms, it would not be a LSNM. Given we are not aware of any identifiable SCMs in the latter two forms, the problem would also become ill-defined if CAREFL were in the latter two forms. > Re: Weakness, Little Discussion of GRCI The key theoretical insights from our work is that: 1. One **does** need to worry about noise misspecifications if a likelihood-based method is used for LSNMs, such as CAREFL-M [11] or LOCI-M [9]. 2. One **does not** need to worry about noise misspecifications if an independence-based method is used for LSNMs. This can be any appropriate independence-based methods for LSNMs, such as the GRCI approach as you say or the proposed CAREFL-H. We provide theoretical analysis on why likelihood-based methods fail, to promote (any) independence-based methods as a solution for LSNMs in these difficult settings, including GRCI and the proposed CAREFL-H. > Re: Questions, Scaling Utilizing independence testing after a likelihood-based method to achieve high accuracy does not imply the likelihood-based method itself does not give flipped likelihoods (low accuracy). For example, on the right side of Table 1 in our paper, CAREFL-H gives high accuracy but CAREFL-M gives flipped likelihood (low accuracy). Please recall that the proposed CAREFL-H method is a combination of the likelihood-based CAREFL-M and subsequent independence testing. The likelihoods of ANMs can flip under correctly specified noise prior, too. This is mentioned on lines 159-161 and summarized in Figure 1 in our paper. Our experiments further support this point in Appendix O (please refer to lines 567-571). We sincerely appreciate the time you've dedicated to the discussion. Thank you very much!
Thank you for correcting my understanding of the function class that CAREFL covers, this was helpful! While the results are limited to the LSNM setting, both the theoretical and empirical contributions have become clearer, and I see their benefits for future work. I am willing to increase my score to 6.
Thank you very much for the positive comments!
Response to the Rebuttal
Thank you for addressing my concerns and questions. I raise my score from 6 to 7.
Helpful rebuttal
Thanks for the further, helpful clarifications. I am happy to keep my original score.
Thank you so much for reviewing our work and the positive comments!
Thank you for your response. Suitability is a very strong assumption that does not follow from the universal approximation theorem since there is no guarantee what function is actually fitted. Additionally, I again repeat the point that testing sine, sigmoid and tanh do not sufficiently support the claim "T is empirically suitable under noise misspecification and misleading CVs". This ties into the main criticism: the authors briefly mention that misspecifying $f$ and $g$ can hurt IT methods, i.e. lack of suitability. The title of the paper suggests a review of up and downsides of both classes, hence, we at least expect empirical simulations on the effect of function misspecification on IT and ML methods, as done for noise misspecification. Overall, even though sentences on this topic are found scattered across the paper, an actual investigation into the pitfalls of IT is missing. The analysis of the pitfalls of IT methods (such as sample complexity) in the rebuttal is appreciated and I most strongly suggest dedicating an entire section of the manuscript that discusses this, including theory and simulations.
Thank you for the follow-up. ### Re: Guarantee of Suitability As mentioned earlier in our previous response and within the paper: - Theoretically, **it is difficult, if not impossible, to guarantee** suitability with neural networks (lines 202-203 in the paper). - Empirically, the universal approximation of neural networks indicates **the capacity of the neural network regressors being suitable** under careful fine-tuning and hyperparameter selection. - Our experiments in Table 2 provide further empirical support for this, including several non-linear functions (tanh, cosine, sine, and sigmoid) and different noise distribution (continuous Bernoulli, uniform, and exponential). - Please also refer to the first global response (Re: Robustness of Independence-Based Methods Under Noise Misspecification in LSNMs) - It explains why independence-based methods outperform likelihood-based methods under noise misspecification in LSNMs. - The robustness of independence-based methods is further supported empirically by our experiments with 580 synthetic and 99 real-world datasets in Section 6. ### Re: Function Misspecification (the Reviewer's Main Criticism) - Misspecifying the functions $f$ and $g$ can hurt not only independence-based methods but also likelihood-based methods. - In fact, nearly all machine learning tasks can suffer in performance due to different levels of function misspecification. - Similar to numerous other studies, the ground-truth functions are unknown and learned by neural networks in our work. - According to the first global response (Re: Robustness of Independence-Based Methods Under Noise Misspecification in LSNMs), **misspecifying the functions $f$ and $g$ does not affect the robustness of independence-based methods as long as the conditional means and standard deviations are approximated**. - As stated in multiple places in the paper (e.g., lines 8-12, 41-49 & 58-59), we focus on investigating **noise misspecification** in LSNMs. Investigation on function misspecification (including empirical simulations on the effect of function misspecification as you say) is out of the scope of this work. ### Re: Up and Downsides of Both Types of Methods & Pitfalls of Independence-Based Methods (Such as Sample Complexity) Referring to a previous response (Re: Question 2. Benefits of using maximum likelihood), the cases where likelihood-based methods outperform independence-based methods are already stated in the paper (lines 155-156 and 255-262). Furthermore, - It is important to note that **our work is not to conduct an exhaustive survey** comparing independence-based and likelihood-based methods in general. We reiterate that **our primary focus is under the settings of noise misspecification** in LSNMs (e.g., lines 8-12 in the abstract, 41-49 & 58-59 in the introduction). A broad-ranging comparison between these two method types in cases other than noise misspecification is out of the scope of this paper. To reiterate, if you refer to "pitfalls" as sample complexity or computational cost, we have addressed them in the paper and in the earlier response (Re: Pitfalls / Downsides of Independence Test). Last but not least, we do not consider sample complexity (or other "pitfalls" you may have in mind) of independence testing to be a substantial criticism against independence-based methods under the settings of interest, because: - We have extensively demonstrated in Table 1 and Section 6 that independence-based methods, **despite their potential for lower sample efficiency or other "pitfalls," outperform** likelihood-based methods across a range of hyperparameter choices and datasets. Likelihood-based methods **fail notably under noise misspecification, even without the "pitfalls"**.
Location-Invariant Noise in Synthetic Data Experiments?
Could the authors please comment on how the data generating process for the experiments in Sec. 6.1 are described in Appendix L, as well as in what way these are conducted on synthetic data with location scaled noise models? Unlike what you write ("ground truth LSNMs") it seems you actually evaluate on *non-location scale noise datasets* by Tagasovska et al. rather than the LS variants? Furthermore, I would disagree with calling the Tübingen cause-effect benchmark dataset 'synthetic' and while I immediately believe that some of its entries may contain location-scaled noise, most certainly not all.
### Re: Data Generation Process in Section 6.1 As stated in the first sentence of Section 6.1 (line 225-226), the data generation process is provided in Appendix L (lines 539 - 545), including: - What LSNM SCMs are used (Equation 10): - $Y := \tanh(X \cdot \theta_1 ) \cdot \theta_2 + e^{\cos(X \cdot \psi_1) \cdot \psi_2 } \cdot Z_Y$ - $Y := \sin(X \cdot \theta_1 ) \cdot \theta_2 + (\tanh(X \cdot \psi) + \phi) \cdot Z_Y$ - $Y := \sigma(X \cdot \theta_1 ) \cdot \theta_2 + \sigma(X \cdot \psi_1) \cdot \psi_2 \cdot Z_Y$ - What noise distributions are used: continuous Bernoulli, uniform, exponential, beta, Gaussian and Laplace. - How parameters (e.g. $\theta, \psi, \phi$) are sampled following previous works. According to Equation 1 of the paper, these SCMs from Equation 10 are LSNMs, or the LS variants as you say. ### Re: Ground-Truth LSNMs are Non-LSNMs The datasets in Section 6.1 are generated by LSNMs and the datasets in Section 6.2 and 6.3 are not. We only used the phrase "ground-truth LSNM" in Section 6.1. The reasons to include these non-LSNMs in our evaluation are as follows: - In Section 6.2 and 6.3, we use the SIM benchmark and the real-world Tübingen cause-effect benchmark, respectively, for evaluation, because numerous previous studies in the domain of LSNMs utilize these two benchmarks for evaluation, e.g. [9, 29, 11, 30, 28]. - Results from both LSNMs and non-LSNMs demonstrate that the proposed method performs well not only when the ground-truth SCMs are strictly LSNMs (i.e. Section 6.1) but also when the ground-truth SCMs are unknown or not strictly LSNMs (i.e. Section 6.3 & 6.2). ### Re: Calling the Tübingen Cause-Effect Benchmark "Synthetic" We have consistently referred to the Tübingen cause-effect benchmark as **"real-world"** throughout the paper (please see lines 52-53, 276). There is no instance in the paper where we refer it as "synthetic".
Re: Contribution (Continued) We would like to provide further clarification regarding the difference between our work and [23]. As we state in the paper (lines 96-97) and in the earlier global response (Re: Contribution), [23] focuses on ANMs (see Equation1 of [23]) and we focus on LSNMs. In Section 4 of [23], the authors discuss the complication of fitting data generated by a ground-truth ANM model with an LSNM model. Since the backward model of an ANM often takes the form of an LSNM, fitting data generated by a ground-truth ANM with an LSNM model may result in an incorrect causal direction. It is important to note that [23] does not address scenarios in which the ground-truth model is an LSNM. Therefore, despite their brief discussion of using LSNMs as the fitting model, the ground-truth model in [23] remains consistently ANMs. In contrast, our ground-truth model is LSNMs. Therefore, to reiterate, the theoretical findings of [23] contribute to ANMs, whereas our theoretical results contribute to the more general LSNMs.
Decision
Accept (poster)