Independent Component Analysis (ICA) aims to recover independent latent variables from observed mixtures thereof. Causal Representation Learning (CRL) aims instead to infer causally related (thus often statistically dependent) latent variables, together with the unknown graph encoding their causal relationships. We introduce an intermediate problem termed Causal Component Analysis (CauCA). CauCA can be viewed as a generalization of ICA, modelling the causal dependence among the latent components, and as a special case of CRL. In contrast to CRL, it presupposes knowledge of the causal graph, focusing solely on learning the unmixing function and the causal mechanisms. Any impossibility results regarding the recovery of the ground truth in CauCA also apply for CRL, while possibility results may serve as a stepping stone for extensions to CRL. We characterize CauCA identifiability from multiple datasets generated through different types of interventions on the latent causal variables. As a corollary, this interventional perspective also leads to new identifiability results for nonlinear ICA -- a special case of CauCA with an empty graph -- requiring strictly fewer datasets than previous results. We introduce a likelihood-based approach using normalizing flows to estimate both the unmixing function and the causal mechanisms, and demonstrate its effectiveness through extensive synthetic experiments in the CauCA and ICA setting.
Paper
Similar papers
Peer review
Summary
The authors propose CauCA as a middle ground between ICA and CRL. They introduce two new assumptions, which is (i) knowledge of the causal variables and how they are related, and (ii) knowledge of the causal variables that were intervened upon to generate each dataset. With these two assumptions, CauCA identifies the latent variables up to element-wise nonlinearity. This is done with a single technical assumption regarding the pre- and post- intervention causal mechanisms, and without the more common assumptions e.g. restrictions on the prior or mixing function. The authors validate their method with a toy problem with synthetic data.
Strengths
The paper is well written, and I appreciate the effort the authors made to intuitively explain their definitions, assumptions, and results, and to contextualize their work in relation to the broader landscape of identifiable representation learning. This made the paper significantly easier to understand. I like the authors' approach of tackling a middle ground between ICA and CRL, where we use domain knowledge of what causal variables are involved, and how they are related. The main strength of this paper is that it provides a strong identifiability result (element-wise nonlinearity, no permutation ambiguity) with few technical assumptions relative to the existing literature. There is also a nice connection to the existing identifiability results in nonlinear ICA / VAE with auxiliary variables (Khemakhem et al., 2020).
Weaknesses
This paper is weakened by a highly unrealistic assumption, which is knowledge of the intervention targets $\tau_k$ for each dataset $\mathcal{D}_k$. I can't see how this assumption could be satisfied on a non-synthetic dataset. Other existing work also make strong assumptions that are arguably unrealistic, but those models can at least be estimated from real-world datasets, and whether they are identifiable or not is a separate question. The fact that the estimation of this model hinges on an unrealistic assumption is a significant limitation in terms of its practicality.
Questions
Can you think of a way to use an existing dataset to demonstrate this method on, without having access to the underlying data generating process? The data does not need to be complex. Using something like dSprites as an example, the dataset would need to be partitioned such that e.g. in $\mathcal{D}_1$, only the shape changes, and in $\mathcal{D}_2$, only the color changes, etc. You need access to the ground truth latents to split the dataset this way, which is what I mean by "requiring access to the underlying data generating process." I will increase my score if I can be convinced that this approach has some practicality.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
As the rest of my review makes clear, I believe that assuming access to $\tau_k$ is a significant limitation of this work, but the authors did not discuss this point in the paper.
Summary
The paper provides identifiability results for causal component analysis, where the causal structure of the latent variables is assumed to be known. The goal is to identify the causal mechanisms and mixing functions up to a certain degree of uncertainty. Compared to the previous work with unknown graphs and linearity, the proposed result is a trade-off between constraints on the structure (known causal graph) and function.
Strengths
1. The paper considers an interesting problem that is important for the development of causal analysis in the context of deep learning. 2. The writing is very clear. 3. The visualization of the violation of Assumption 4.1 is particularly helpful.
Weaknesses
My comments are as follows. They are not necessarily weaknesses since some of them could be further clarified during the rebuttal. 1. If the causal graph is given, it seems that the considered setting could possibly be transferred to the traditional one where there is no dependence among latent variables but only an unknown mixing process between latent and observed variables. Since interventional data is a specific type of multi-environment data (with an additional indicator for interventions), the novelty compared to the previous results (e.g., [1] and many others) does not appear to be fully clear. This has been partially discussed in Sec. 4.2 but it seems that the principal difference and connection, in both setting and proof techniques, should also be discussed deeper in other parts of the paper. The required number of environments is strictly lower but at the cost of interventional data. 2. To justify the assumption on the known causal graph, the paper compared the spaces of potential DAGs and spurious solutions for the representation, and suggested that assuming the known structure is reasonable since 'there are only finitely many possible DAGs'. IMHO, it seems that the statement is not strong enough to support the assumption, since the exponential nature of the DAG space, although finite, is still unacceptable in most scenarios. 3. With imperfect intervention, the ambiguity of the identification is a mixture of nodes that are ancestors of the target node. While this is a non-trivial reduction from the unconstrained space, it remains uncertain about the significance of this result. In the considered setting (i.e., there are no edges among observed variables), only the ancestors of each node can have causal effects on the target node. Thus, the ambiguity does not seem to be surprising with known causal graphs. 4. Assuming a known causal graph is a relatively strong assumption. Since the considered setting is nonparametric, causal discovery is a challenging task. Thus, more discussion on the assumption of a known causal graph would be helpful to support the task. [1] Hyvarinen, A., Sasaki, H., and Turner, R. (2019). Nonlinear ICA using auxiliary variables and generalized contrastive learning.
Questions
Apart from the comments above, here are some questions: 1. Are there any noises considered during the generating process or it is fully deterministic? 2. If single-node interventions can be performed in practice, in most cases, doesn't this mean that the ground-truth latent variables have already been taken as a prior, especially with the known causal graph?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
It would be better if a separate discussion of limitations could be included.
Summary
The authors consider a class of settings in the realm of causal representation learning (CRL) and study their identifiability. More specifically, they consider Causal Component Analysis where the latent causal graph of the generative model is known and interventions are performed. As the authors show, the problem is still not trivial - the mixing function and the mechanisms are to be learned - and identifiability only up to some transformation is possible. The authors prove identifiability for several different cases: the type of interventions performed (perfect vs imperfect, single-node vs fat-hand) and the structure of the latent causal graph (empty vs non-empty). Finally, the authors corroborate their theoretical findings by performing experiments with synthetic data.
Strengths
The article is clear and well-written. It is original as it addresses a slightly different setting than "classical" CRL. The results can have an impact on CRL research since they show in easier cases what is the best form of identifiability one can achieve.
Weaknesses
One limitation is the fact that the settings considered are extremely artificial and contain a lot of knowledge about the problem at hand. The latent graph is known and interventions are performed on almost all latent variables. So, compared to other CRL works, the assumptions are too strong to have practical applications. However, the goal of the authors seems more to have identifiability that gives a kind of "upper bound" (i.e., the identifiability in this simpler setting is the best we can do) for actual CRL that considers weaker assumptions. Another weaker part is the experiments. I understand that there are toy experiments to validate the identifiability results, but I am not sure that the baselines are that interesting: linear and empty graphs. Could it be compared to another method that learns the graph? Some results could be interpreted or explained in more detail (see the Question section).
Questions
### Questions - Surprisingly in the results presented in Fig.4 a) and e) the empty graph, which has no information about the latent causal graph, seems pretty competitive with CauCA. Any thoughts on why it is the case? - Also, any intuition why having more non-linearity would lead to a lower MCC (Fig 4.c)? - Another interesting parameter to investigate would be the edge density of the latent graph. Have you considered this experiment? ### Typo: - line 72: "leads extensions", words are missing - line 98: "the" is repeated - Fig 2, caption: "As can be seen", words are missing - Some punctuation marks are missing in the equation (e.g., Eq.5) - line 193: "untintervened" - Fig 4.c, "No. nonliniearities" - Eq. 9, the second term of LHS is missing a log. - Suggestion: I would specify the type of normalizing flow employed in the main text (I know it is in the appendix).
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
While the assumption are clearly defined in the different theoretical results, the limitation could be more explicitly presented (see my first point in the weakness). Societal impact is not applicable.
Summary
This paper introduces the concept of Causal Component Analysis (CauCA), which lies between Independent Component Analysis (ICA) and Causal Representation Learning (CRL). CauCA generalizes ICA by considering the causal dependence among latent components while assuming knowledge of the causal graph, focusing on learning the unmixing function and causal mechanisms. The paper establishes identifiability results for CauCA based on multiple datasets generated through interventions on the latent causal variables, leading to new identifiability findings for nonlinear ICA. It also proposes a likelihood-based approach using normalizing flows to estimate the unmixing function and causal mechanisms, demonstrating its efficacy through extensive synthetic experiments in both CauCA and ICA settings.
Strengths
Strengths of this paper include a comprehensive presentation of Causal Component Analysis (CauCA) and its connections to existing methods such as Independent Component Analysis (ICA) and Causal Representation Learning (CRL). The paper thoroughly characterizes the identifiability of CauCA from multiple datasets, considering various types of interventions on latent causal variables, even in the nonlinear and nonparametric case. By leveraging the modularity of causal relationships, the paper introduces a novel perspective to tackle nonlinear ICA, leading to new identifiability results that require fewer datasets for the same level of identifiability. The paper also proposes an estimation procedure based on normalizing flows, and extensive synthetic experiments in both CauCA and ICA settings demonstrate the effectiveness of this approach in accurately recovering the latent causal components.
Weaknesses
None.
Questions
None.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
The authors did not discuss the limitations, so one paragraph on this may need to be added.
I appreciate the comprehensive response provided by the reviewers, which effectively addressed my concerns. I am also thankful for the inclusion of additional experiments. I believe this paper should be accepted.
Thank you for your response. I have read the other reviews and rebuttals and will keep my original score.
Thank you for your detailed response, and for developing the additional theory on different degrees of knowledge about the intervention targets. I was mainly interested in the final paragraph of your response, and I actually did think about single cell genomics as a potential application, since the intervention targets are known. However, in this setting it's unrealistic to assume knowledge of the variables that govern the biological processes of interest. I still can't think of any non-toy settings where both assumptions are met. Since my personal bias is that we should be advancing methods that have potential for practical applicability, I am maintaining my score.
Thank you for your response. This work is not necessarily focused on developing new practical methods, but rather on understanding what is in principle possible (identifiability) and impossible in CauCA or CRL. The setting in our paper (knowing intervention targets up to graph automorphisms) is strictly included in the setting where less knowledge about targets are given (i.e. unknown targets with every node intervened on at least once), thus our results are a special but unavoidable case in the proof for the latter setting in the future. While we agree, that the method and its underlying assumptions are not very practical in its current form, we still believe (and reviewers xM6s and nsXw seem to agree) that our theoretical results---despite the personal bias of the reviewer---still constitute a valuable contribution, whose insights may help construct more practical methods in future work.
Response to author rebuttal
Dear reviewer, The author rebuttal appears to have presented several targeted responses to your questions. Are your questions appropriately addressed? If they are, would you consider re-assessing your score in light of them. If not, please do provide additional context and feedback to the author. In either case, please provide an acknowledgement of the effort the authors put in, why your questions have (or have not) been addressed and what your assessment of the work is in light of this evidence with a view to reach consensus with the other reviewers on this work. -AC
Thank you for the detailed responses, and thanks AC for the reminder. After reading the responses and considering discussions from other reviews, I still have several questions about my concerns. Of course, please let me know if I have misunderstood anything. - When considering the transfer of the given setting to the original ICA problems with a known structure, I'm referring to a process like the following: Suppose we have [{z2} -> x1, {z1} -> x2, z1->z2] as the ground-truth structure in CauCA. It appears we can directly transfer this to [{z1, z2} -> x1, {z1} -> x2] since all observed variables influenced by z2 are also influenced by z1 (given z1->z2). Earlier works have addressed the identifiability of latent variables by assuming distributional changes, which includes the introduction of interventional data. Since the proposed method doesn't aim to uncover the hidden structure among latent variables (because it's provided as an assumption), the principal difference is not very clear to me. - Considering that the ultimate goal of identifiability theory is component-wise identifiability, the "known intervention targets" assumption seems somewhat akin to disentangling individual latent variables before the estimation process. If we already know the targets of single-node interventions, these latent variables appear to be pre-disentangled. - In addition to the lacking of application scenarios discussed by other reviewers, I'm not fully convinced of its theoretical significance, especially when compared to existing works on nonlinear ICA. The combined assumptions of known structures and known intervention targets seem relatively unrealistic. They provide a substantial amount of information about the hidden data-generating process, especially when indeterminacies like element-wise rescaling persist after estimation. It feels like too much information is pre-supplied in the form of assumptions. Specifically, we would like to recover the generating process both qualitatively and quantitatively. However, in the proposed setting, the information from both perspectives is either given as assumptions or not possible to recover. From a qualitative perspective, the structural information is provided through known graphs, and the individual latent variables are pre-disentangled, allowing us to ascertain the targets of single-node interventions prior to estimation. Quantitatively, the indeterminacies introduced by element-wise rescaling render the recovery of the effect theoretically impossible.
Thank you for engaging in a discussion of our work. We address your remaining questions in our reply below. Please do not hesitate to ask in case you have additional questions. > “Suppose we have [{z2} -> x1, {z1} -> x2, z1->z2] as the ground-truth structure in CauCA. It appears we can directly transfer this to [{z1, z2} -> x1, {z1} -> x2] since all observed variables influenced by z2 are also influenced by z1 (given z1->z2).” In a latent CBN model (Def. 3.1), the graph $G$ only describes the causal relationships among the latent variables $\mathbf{Z}$: **it contains no information on the functional relationship between the $\mathbf{Z}$ and the $\mathbf{X}$ variables**, which is described by the unknown mixing function $\mathbf{f}$. Moreover, **$\mathbf{f}: (z_1, z_2) \mapsto (x_1, x_2)$ maps all variables in $\mathbf{Z}$ to all variables in $\mathbf{X}$**. The objective we focus on is _blind source separation_: i.e., reconstructing (up to unavoidable ambiguities) the sources $z_1, z_2$ based on the observed mixtures $x_1, x_2$. The mixing described in your example can be considered as a trivial case in our setting, since the sources $z_1, z_2$ would already be “separated” in the observed components $x_1, x_2$. Conversely, in our work, we allow $\mathbf{f}$ to be any diffeomorphism: this matches the generative model of nonlinear ICA and many works on CRL such as [1][2][3][4]. We hope that our explanation above also clarifies that the example you provided cannot be considered as a reduction of CauCA to the ICA setting, since, in the setting you described, the variables $z_1$ and $z_2$ are not independent, and the mixing is “trivial”. As we explained in the first paragraph of our rebuttal, we believe that such a reduction is impossible. > “Earlier works have addressed the identifiability of latent variables by assuming distributional changes, which includes the introduction of interventional data. Since the proposed method doesn't aim to uncover the hidden structure among latent variables (because it's provided as an assumption), the principal difference is not very clear to me.” To explain the contributions of CauCA, we can refer to the two closest lines of work. (1) The first one is interventional CRL, which differs from CauCA by considering an **unknown** latent graph, and where identifiability results are based on strong assumptions on either the mixing (e.g., linear [5]) or the latent SCMs (e.g., linear [2]); and (2) nonlinear ICA with auxiliary variables, where the mixing is nonlinear and the graph is **known and trivial** (i.e., empty, with no arrows). The difference is that in our work we consider a **known but nontrivial graph**, and a **nonlinear** mixing and latent SCM. The extra information on known (single or block) targets up to graph automorphism allows us to trade off the complexity coming from considering nontrivial graphs, as opposed to ICA. Therefore, CauCA cannot be reduced to any of the results in nonlinear ICA (see the answer above), and is not a special case of any existing results in interventional CRL. > “Considering that the ultimate goal of identifiability theory is component-wise identifiability, the "known intervention targets" assumption seems somewhat akin to disentangling individual latent variables before the estimation process. If we already know the targets of single-node interventions, these latent variables appear to be pre-disentangled.” To show that the latent variables are not pre-disentangled in this setting, we refer to Fig.2 in the manuscript (we were happy to read that you found it “particularly helpful”): without assumption 4.1, even if we already know the targets of single-node interventions in the latent model $(\mathbf{f}, (\mathbb{P}^k_Z)_{k\in \{0, \ldots, d\}})$, there are still uncountably many spurious solutions $(\mathbf{f}’, (\mathbb{Q}^k_Z)_{k\in \{0, \ldots, d\}})$ such that $\mathbf{f} \mathbb{P}^k = \mathbf{f}’ \mathbb{Q}^k$ $ \forall k \in \{0, \ldots, d\}$ while $\mathbf{f}’^{-1} \circ \mathbf{f}$ is not an elementwise scaling. This shows that disentangling the latent variables is not trivial even with known intervention targets. > “Quantitatively, the indeterminacies introduced by element-wise rescaling render the recovery of the effect theoretically impossible.” Our Lemma 3.3 proves that the ambiguity up to element-wise scaling is unavoidable in CauCA if no constraints exist in the latent distributions. Therefore, identifiable up to element-wise monotonic scaling is the best one can hope for. Notice that in those ICA/CRL papers where the mixing function is any diffeomorphism, all the identifiability results allow ambiguities up to elementwise scalings: [1][2][3][4], etc. Even in Thm3 of [1] and Thm1 of [4], identifiability up to affine transformation of the $nk$ sufficient statistics in an exponential model is in fact still a *nonlinear* ambiguity of $\mathbf{f}$.
One could criticize that the assumption of a known graph is unrealistic. Viewed as a special case of our method, ICA is also making an unrealistic assumption: that of a trivial graph. This analogy might lead you to think that our methods should somehow be reducible (modulo a different graph) to those known from nonlinear ICA. However, in CauCA, we suppose the distributions are **Markov** to a given graph $G$, but not necessarily **faithful** to $G$. This implies that independent components are allowed in any CauCA model $(G,\mathcal{F}, \mathcal{P}_G)$. Since the independent component distributions are a null subset in the set of distributions Markov to a nontrivial G, CauCA is a strictly harder problem than ICA. To further clarify the difference between CauCA and nonlinear ICA, we will detail how our proofs differ from the ones in nonlinear ICA. We thus far omitted this for conciseness, but we will be happy to elaborate on this in the Appendix. For the special cases of CauCA where there are no edges in the causal graph, corresponding to ICA, Thm.1 of [1] proves identifiability with (in our terminology) $2d$ interventional environments of unknown intervention targets ($d$ is the number of nodes). Our identifiability results can then be compared with Thm. 1 of [1] in two cases: i) **Trivial graph—Previous work:** A core step in the proof of Thm. 1 of [1] is in the last two equations on page 23 of CauCA (which is essentially the same as eq.(20,21) of [1]). We will refer to the proof of Prop. 4.7 of CauCA in the following (since the proof of _this proposition specifically_ is indeed similar to the one of Thm.1 of [1]): we take twice the partial derivatives of the equation below eq.(30) of CauCA, i.e., we calculate the Hessian matrix of the two sides. Then one can obtain a system of $2d$ equations, and identifiability corresponds to the uniqueness of the solution of the system $\mathbf{0}=\mathbf{A}\mathbf{x}$, where $\mathbf{A}$ is $\mathbb{R}^{2d \times 2d}$. So $\mathbf{A}$ needs to be invertible—i.e., the “variability” assumption. **Our work:** Our Prop. 4.4 and Thm. 4.6 show that if we know the intervention targets, then _only the Jacobian_ of the above equation, instead of Hessian, is required, and thus the linear system will become $d$ equations instead of $2d$. ii) **Nontrivial graph:** Our proof of Thm. 4.2 is even farther from all ICA proof techniques, including [1]. A peculiarity of ICA is that, after taking the Hessian, the LHS of the last equation on page 23 of CauCA becomes zero. Unfortunately, with a nontrivial graph, it is impossible to get $0$ after taking the partial derivatives of every coordinate of $z$, c.f. eq. (20) of our paper. All the techniques of nonlinear ICA rely on this zero LHS over every coordinate of $z$. **Therefore, all the proofs in our work are indeed different and irreducible to Thm. 1 of [1]**---with the exception of Prop. 4.7, which might have caused the confusion. [1] Hyvarinen, A. et al. (2019). Nonlinear ICA using auxiliary variables and generalized contrastive learning. [2] Buchholz, S. et al.. (2023). Learning Linear Causal Representations from Interventions under General Nonlinear Mixing. [3] Brehmer, J., et al. (2022). Weakly supervised causal representation learning. [4] Khemakhem, I. et al. (2020). Variational autoencoders and nonlinear ICA: A unifying framework. [5] Seigal, A et al. (2022) Linear causal disentanglement via interventions.
Decision
Accept (poster)