Score matching through the roof: linear, nonlinear, and latent variables causal discovery

Summary

This paper present novel theoretical results to identify causal effects in restricted ANMs even in case of unobserved confounders.

Strengths

**The paper provides novel contributions to the field of score-based causal discovery by extending previous works to confounded restricted ANMs. Based on these contributions, I strongly support this paper's acceptance.** - the problem definition is very useful to orient readers - the theoretical results are novel - the experiments compare against a sufficient number of baselines, and though the proposed method is not SOTA, it compares well and has better theoretical guarantees Based on the authors' response, I am eager to improve my score.

Weaknesses

I have a few remarks on improving the flow of the paper; however, even the first four points are not considered major issues. - Even though condition 1 is a well-known result in the causality literature, I suggest explaining why that admits linear models and including some description of **restricted ANMs** in the main text (at least for me, it is not evident, especially since the condition lacks intuition). To be clear, even this point does not diminish the main contribution, which I see regarding the results for confounders. - As **inducing paths** are an important concept for the main contributions, please _include it in the main text_ if space permits (suggestion: you can reduce spacing in $\texttt{itemize}$ by setting $\texttt{\\\\begin\\{itemize\\}[nolistsep]}$ ) - I could **not find the definition of an active path** (not even in Def. 5, where it is said to be defined); I presume it is a path that is not blocked, but it would be better to state this explicitly. Maybe it would even be better to use "a path that is not blocked" instead of introducing new terminology (this is the first time I encountered "active paths"; I could be wrong about this) - The text is sometimes difficult to follow, due to heavy reliance on notation. I'd consider delegating the not crucial part to the appendix (potential candidates in 2.1) and using the remaining space to explain the main quantities better, especially the residuals (e.g., Eq. 12) ## Minor points - please specify what you are calculating the expectation with respect to (using a bold E is also unconventional, though it's clear from the context it's an expectation) - as the mathematical objects for d-/m-separation are distinguishable, you might consider dropping the superscript to simplify notation; also, I'd suggest adding whitespace after $\perp^d_\mathcal{G}$ and the like to make it easier for the reader to attribute the indices to $\perp$ and not the not on its right - I could not find the definition for $\dot{\cup}$ - in the explanation of Prop 3., the wording makes it a bit hard to discern that you also provide intuition for the second part; it would help if you refer to _Part (ii)_ explicitly - _Score matching through the roof_ in the title does not have added value for me, I'd consider rephrasing it to convey the message that "we propose score-based causal discovery methods for confounded restricted ANMs"

Questions

Some methods in the literature [1-4] use the Jacobian of a learned neural network instead of relying on the score for causal discovery/identifiability claims. How do you see the relation of your contribution to those Jacobian-based methods? - [1] Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient-based neural DAG learning. In 8th International Conference on Learning Representations, ICLR 2020 - [2] Lazar Atanackovic, Alexander Tong, Jason Hartford, Leo J. Lee, Bo Wang, and Yoshua Bengio. DynGFN: Bayesian Dynamic Causal Discovery using Generative Flow Networks, February 2023. - [3] Patrik Reizinger, Yash Sharma, Matthias Bethge, Bernhard Scholkopf, Ferenc Huszar, and Wieland Brendel. Jacobian-based Causal Discovery with Nonlinear ICA. Transactions on Machine Learning Research, April 2023. - [4] Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, Antti Kerminen, and Michael Jordan. A Linear NonGaussian Acyclic Model for Causal Discovery. Journal of Machine Learning Research, 7(10), 2006.

Rating

8

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

The authors provide an **honest comparison** of their method in the experiments, clearly stating its limitations compared to other methods.

Summary

The authors propose AdaScore, a method for causal discovery that generalizes previous work based on score matching for SCMs with possibly latent nodes. They combine connections of the score to conditional independence as well as to additive noise SCMs and show that a NoGAM-type procedure works to recover the direction of non-confounded edges of the corresponding partial ancestral graph.

Strengths

- Adapting NoGAM [1] to the case allowing hidden variables is a practically meaningful contribution. - Model assumptions are somewhat weakened compared to CAM-UV by allowing for general mechanisms within blocks of observed and latent parents.

Weaknesses

The novelty of the paper lies primarily in the application of NoGAM to orient very specific edges in a partial ancestral graph (PAG) (the ancestral graph that represents the Markov equivalence class, in analogue to a CPDAG). This falls significantly short of the main contributions as described by the authors on l33--55. Specifically: - The authors state that they show how constraints on the Jacobian of the score can be used as conditional independence testing. However, the extent to which this is done is only by noticing the equivalence between conditional independence and the corresponding zero in the Jacobian term (previously noted in [1,2]), without any formal analysis of the proposed t-test (Appendix C) as a statistical test of conditional independence (which happens to be a notoriously difficult test). - The authors state that their identification results for additive noise models generalize the previous results obtained by previous works. In l193, the authors state "we remove the nonlinearity assumption (of [3]) and make the weaker hypothesis of a restricted additive noise model", but 1) this is a stronger assumption than additive noise, not a weaker one, and 2) the authors in [3] also consider the same restricted additive noise model. - The authors claim that AdaScore is able to handle a broad class of causal models (l54), but three out of four possible situations are direct applications of existing work. 1) Under no structural assumptions with or without latent confounders, AdaScore simply performs constraint-based causal discovery (FCI) using the conditional independence properties of the Jacobian of the score, a straightforward application of [1] also previously noticed in [2]. 2) Under an additive noise assumption, AdaScore is exactly equivalent to NoGAM. 3) Only under an additive noise assumption with hidden confounders, does AdaScore generalize NoGAM to orient unconfounded edges of the PAG returned by FCI, which may be very few of the discovered adjacencies. ### Other comments - The experiments do not seem to suggest that AdaScore out performs other methods in any meaningful way---in fact, in Figure 1 a) AdaScore is completely equivalent to NoGAM, and is thus redundant. In Figure 1 b), where AdaScore should distinguish itself, it does not appear to be consistently better than CAM-UV. - Much of the paper (> 5 pages) is spent on directly describing previous works, NoGAM[3] and/or provide basic background on DAGs and MAGs. [1] Spantini et al., "Inference via low-dimensional couplings." JMLR 2018. [2] Montagna et al., "Scalable causal discovery with score matching." CLeaR 2023. [3] Montagna et al., "Causal discovery with score matching on additive models with arbitrary noise." CLeaR 2023.

Questions

- Is it correct to say that the unconfounded edges that are oriented by AdaScore are the purely _undirected_ edges of the PAG, and not the potentially bidirected ones?

Rating

3

Confidence

4

Soundness

3

Presentation

3

Contribution

1

Limitations

The authors do not adequately discuss the limitations of their method---the limitations section in the appendix focuses purely on the empirical study. The authors claim that AdaScore is adaptive in the sense of being "less reliant on prior assumptions which are often untestable", but this is only in the sense that it performs different algorithms depending on user specification, which hardly constitutes one single unifying adaptive algorithm.

Summary

The paper extends theoretical results about causal discovery through score matching to encompass both linear and non-linear SCMs and lift the sufficiency assumption. The theoretical results relax the non-linearity assumption of Montagna et al 2023 by swapping it with the less restrictive one of restricted ANM from Peters et al 2009. As for the latent confounder detection a parallel with m-separation is drawn using results from Spantini et al 2018, to establish that the score will be non-zero in the presence of an active path. Following the theoretical results, an algorithm to estimate causal graphs from data is proposed and evaluated, generalizing the NoGAM algorithm of Montagna et al 2023, which only covers the non-linear case.

Strengths

The paper is clearly written and, if it wasn’t for some of the definitions relegated to the appendix, very easy to follow. The theoretical results, particularly Propositions 2 and 3 are important extensions of the score-matching methodology for causal discovery, dealing with both the non-linearity and sufficiency assumptions of the method proposed in Montagna et al. 2023.

Weaknesses

The paper motivation is basically the weakening of current assumptions for causal discovery methods. However, assumptions and benefits of the proposed methodology are not clearly specified. In line 74, the authors state that faithfulness is assumed (I believe, it is kind of hidden in the background notions). If that is the case, the method adopts the same assumption as FCI, plus ANM. So the proposed method is relaxing assumptions compared to CAM-UV, RCD and NoGAM, but adding onto FCI. Regarding the benefits, the alleged flexibility of the method to output DAGs, MECs, MAGs, PAGs, which should make it preferable to FCI, is merely touched upon in the contributions and the experiment section. Proposition 1 is a rather trivial application of the more general lemma in Spantini et al. and it does not specify the required faithfulness assumption to obtain the result from Eq. 6 in the paper. The experimental results show limited added value according to the one metric chosen (SHD) in a synthetic setting. They are not comprehensive enough, with no application to common (pseudo-)real benchmarks (e.g. from bnlearn). More experiments and more metrics are needed as, as it stands, the proposed method seems to add no real value compared to the baselines. Additionally, it is not clear from the experiments if it is really able to identify confounders. Breaking down precision and recall by mark would show this. FCI and a random baseline should be also added for reference. Experiments are conducted on data with at most 9 variables, and the scalability of the method is not shown nor discussed. The model used to estimate residuals is not discussed, nor the assumption that the chosen model fits the data adequately to correctly estimate residuals, and what is needed to assess this.

Questions

- Line 28: FCI weakness is highlighted to be that it outputs PAGs, does the proposed method improve on this at parity of assumptions? - Line 44: ANM is an assumption on the causal mechanism, is it not? - Line 217: "the noise terms recentered by the latent causal effects" what does this mean? - Line 221: "N_i are assumed to be non-Gaussian when f_i is linear" This reduces to RCD, right? - Line 257: "the remaining arrowheads … are identified no better than in the equivalence class" what does this mean? - Line 279: "using hypothesis testing to find vanishing MSE..." how is this set up? - Line 280: how much added value does pruning have? E.g. CAM pruning can have huge impact https://arxiv.org/pdf/1310.1533 - Alg. 1: do you start from a disconnected graph? Where do the "remaining" bidirected edges come from, do you reduce them prior to this? - Line 292: exogenous variables are selected at random, hence could be sink nodes with no consequence on the discovery process. Alternative and more meaningful testing strategies could be to select confounders (e.g. https://proceedings.neurips.cc/paper_files/paper/2021/file/144a3f71a03ab7c4f46f9656608efdb2-Paper.pdf) - Line 305: "presents better performance" I am not sure about this, they seem pretty aligned considering quartiles. Statistical tests to ascertain this are missing. - Line 315: some realistic scenarios are available through e.g. the bnlearn repository (https://www.bnlearn.com/bnrepository/). Any reason not to test on some of the data available there? Minor points and typos - Line 73: reference for d-separation is missing - Line 79: reference for MEC is missing - Line 130: have -> has - Line 147: "directed global Markov property": not introduced. I guess it is just the global Markov condition introduced before? - Footnote 1: provides -> provide - Line 162: "unobserved active paths" are not defined in Definition 5, m-separation. - Line 209: full stop missing - Line 236: the PAG -> a PAG - Line 267: "Appendix C.2" what should the reader expect to find in there? - Line 296: "We fix the […] level" significance

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

yes