Identifiability Guarantees for Causal Disentanglement from Soft Interventions

Summary

This paper focuses on the problem of causal discovery with unobserved causal variables. The authors show that under necessary assumptions, the latent causal model can be recovered by iteratively identifying the interventions that target the source nodes and removing these source nodes. Based on this finding, the authors propose a VAE-based generative model for recovering the causal mechanisms from observational and interventional data. The model shows promising performance on a gene mutation dataset in terms of capturing the interventional distribution and learning the causal DAG between target genes, which demonstrates the effectiveness of the proposed method.

Strengths

• The authors clearly list the steps of identifying the ancestral relationships of nodes and direct edges in a causal DAG with soft interventions and give concrete examples to help readers better understand the assumptions. • The presented experimental results show the proposed model’s capability of learning the causal mechanism in the training data as well as generalizing to unseen interventional data. • The paper is organized and presented in good shape.

Weaknesses

• It is better to briefly discuss the meaning of interventional faithfulness in previous studies. Does it simply mean “all causal variables are observed”? • I think encourage the authors to discuss the validity of assumptions 2 and 3, especially assumption 2, in real-world scenarios as they claim that these are stronger versions of faithfulness assumptions. Minor comments: • In the equation on line 278, should the subscripts be $\hat{I} \notin \mathcal{I}$ and $\hat{I} \in \mathcal{I}$ instead of $I \notin \mathcal{I}$ and $I \in \mathcal{I}$? • What does $s_i(\cdot)$ represent on line 297, the causal mechanism corresponding to the $i^{th}$ node in the causal DAG? • The fonts in Figure 4 are too small. It might be better to change the orientation of the diagram (e.g. using a left-right flow instead of a top-down flow) and make the fonts larger.

Questions

• Why do the authors choose MMD over Wasserstein distance for the distance measure between the generated and true interventional distribution? • Can the proposed method handle the case where there exist **imbalanced interventions**? For example, there is only one sample in each of $\mathcal{D}^1, …, \mathcal{D}^{K-1}$ and 1000 samples in $\mathcal{D}^K$. I expect this situation to be possible in biological data since some genetic mutations can be extremely rare while some are frequently observed. • Besides biological settings, can authors think of other scenarios where the proposed model can be applied?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

The limitations are appropriately discussed. I’m not aware of any direct negative societal impacts of this work.

Summary

This paper studies the task of learning causal variables from high-dimensional observations. This is done under the assumption that single-node, soft interventions are available for all causal variables, as well as the observation function being a full row rank polynomial, e.g. linear. Based on prior work, the paper shows that this reduces to identifying causal variables from a linear combination, and determines challenges to identify the causal variables further in interventional distributions. The paper derives a result to identify causal variables up to linear combinations of its ancestors, and for identifying the causal graph among them. Finally, a practical algorithm, CausalDiscrepancyVAE, is derived and evaluated on genetic data with CRISPR interventions.

Strengths

The paper is overall well written. The paper uses a clear and consistent notation throughout the paper. The setting is well introduced. Further, examples are provided to guide the understanding of the theory. The closest related works for the presented paper are clearly discussed and put into perspective with the goal of this paper. The paper presents both theoretical and empirical results. The theoretical results are backed up by detailed proofs in the appendix. The proofs appear sound and the results are intuitive, but I have not checked all proofs in detail. The setting of unpaired observations with causal relations between variables is challenging. Hence, while not ideal, an assumption for single-node intervention on all variables is reasonable, considering the current stage of causal representation learning (CRL) in general. The empirical results are based on a newly proposed algorithm, that takes advantage of a variational autoencoder setup with additional losses for guidance. Another strength of the paper is its application to real-world data, specifically gene expression data. Many CRL works currently work on purely synthetic benchmarks, which this paper tries to go a bit beyond. The results on the genetic data are encouraging, suggesting that future work in CRL can become potentially used in this application domain.

Weaknesses

The main focus of the paper is its theoretical identifiability results. However, for that, the theoretical contribution of this work is quite limited. The main task of the identifiability theory is to determine the causal variables $U$ from a linear mixing $X=U\Lambda +b$. The problem with identifying $U$ from interventions is then discussed as the effect of interventions potentially being canceled out by linear combinations of the causal variables. Assumption 2 is then stated as simply ruling out this case. However, this assumption is done without much of simplification of this linear combination statement and requires checking all possible linear combinations in $P(U\_j+U\_SC^T)\neq P^I(U\_j+U\_SC^T)$. This is very likely difficult or impossible to verify in any real-world setting given that (a) the causal variables are not known, and (b) one needs to search over all $C$. Similarly, with the identifiability of the graph $G$, the assumption that it is the sparsest within its transitive closure is relatively restrictive and difficult to verify in real-world applications. Finally, it's unfortunate to see that the causal variables are then only identified up to linear combinations of its ancestors. The paper would be strengthened in discussing whether hard interventions can give stronger identifiability in these cases, which is likely the case as in [Lippe et al., 2023]. The proposed model architecture is quite involved and has several components (encoder, decoder, DSCM, MMD, intervention encoder) and hyperparameters ($\lambda, \alpha, \beta$, temperature of intervention encoder, MMD kernels) interacting. The tuning of the hyperparameters is not well discussed and it is only mentioned that delayed linear schedules are used for both $\alpha$ and $\beta$. It is unclear how sensitive the model is to those, and how much tuning is necessary. No ablation study on individual elements of the architecture have been performed, overall making it unclear what each component is contributing and whether parts are necessary. Overall, with such an involved setup without ablation studies, it is unlikely that the architecture will be adopted in future work. Moreover, the theory has as integral assumption that the mixing function is limited to full row rank polynomial (Assumption 1). Nonetheless, the practical algorithm itself uses non-linear neural networks covering a much larger space of possible mixing functions. This appears to violate the assumptions and break its relation to the theoretical results. A discussion on how this affects the algorithm or intended datasets is largely missing. In terms of empirical evaluation, it is great to see experiments on real-world data. However, no baselines like a simple VAE are compared to. This makes it difficult to put the results into perspective. A synthetic dataset on which the identifiability of the causal variables and baselines can be precisely compared is crucial to gain insights in the proposed architecture. Further, considering other modalities like images would greatly improve the range of the empirical evaluation. Finally, it is unclear if the experiments have been performed for multiple seeds, and how the trained models vary over the seeds. Regarding the writing and presentation of the paper, it is overall well written. Still, it clearly targets researchers in the CRL community and is likely difficult to fully understand for researchers outside. In Section 2, 3 and 4.1, it is not always clear which parts are novel for this paper and which builds up on previous results. Overall, assumptions like almost-linear mixing function and single-node interventions on all variables should be clearly stated in the introduction. Additionally, the term 'unobserved causal variables' is confusing with latent variables in common causal discovery tasks, e.g. latent confounders. In this setup, the information of the causal variables is still observed through a mixing function, just the separation and individual variables are not identified. This should be clarified when using this term, e.g. in the introduction (line 52). ### Typos - Line 306: 'to to' ### References [Lippe et al., 2023] Lippe, P., Magliacane, S., Löwe, S., Asano, Y. M., Cohen, T., & Gavves, E. (2023). Causal representation learning for instantaneous and temporal effects in interactive systems. In The Eleventh International Conference on Learning Representations.

Questions

### Review Summary The paper presents an identifiability result for causal representation learning from soft interventions. It's strengths includes the detailed proofs for the presented theoretical results and its application to real-world datasets. However, the theoretical contribution is limited, considering its basis on prior works and extensions on them. The empirical evaluation is further insufficient, requiring more experiments to validate the claims and components of the architecture. Thus, I consider the current version of the paper below the acceptance threshold. ### Questions Main aspects: * How would you verify Assumption 2 and 3 in real-world settings? * Do you have theoretical results on whether hard interventions strengthening your identifiability class or not? * How does your practical algorithm relate to the theory given that the mixing functions have different classes? * Do you have ablation studies on all components of your architecture? Minor points: * Example 1, Line 190: the interventional distribution of $U\_2$ is presented as identical to its observational distribution. I assume this is a typo where the interventional distribution should be $P(U\_2|U\_1)=\mathcal{N}(U\_1+1,1)$ instead? If not, can you explain how the interventional faithfulness is satisfied despite an intervention having no effect? * Line 222: why do you consider here $p$ interventional settings ($I\in \{I\_1,...,I\_p\}$) instead of $K$ ($I\in \{I\_1,...,I\_K\}$) as in Sec. 2 before? --- ### Post-rebuttal update The rebuttal addressed some of my concerns. Considering an improved presentation as discussed in the rebuttal, and the main contribution of this paper being theoretical, I raise my score to '5: Borderline Accept'.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Limitations of the theoretical setting have been shortly discussed in the conclusion.

Summary

This paper proves identifiability of causal disentaglement in latent space in the presence of interventional data. In more details, consider data X generated as X = f(U) where the distribution of U encodes a causal graph G (or a Bayesian network), identifiability is the question of whether f and U can be recovered uniquely. This is known to be not true in general but recent works have shown that this is possible under additional information or biases. This work assumes access to extra data of the form X' = f(U') is available where U' is a single-node soft intervention on U and under various other assumptions shows that identifiability holds. Specifically, if we assume 1. f is a full-rank polynomial 2. the causal variables are linear interventional faithful (which means interventional effets don't cancel each other out), then this work shows that the intervention targets and the transitive closure of the causal graph can be recovered up to an equivalence class. Additionally, with 3. an adjacency faithfulness condition, it's shown that the underlying causal graph can also be recovered up to an equivalence class. Identifiability is a statistically desirable notion that a unique latent variable model could have generated the data. A flurry of recent works have studied identifiability of latent causal graphs under interventions and this work continues this line of work for soft interventions under polynomial mixing. For the proof, the work by Ahuja et al. 2022 is first applied to directly recover the polynomial f up to a linear transformation. What follows next is the main contribution of this work, which is a sort of a peeling procedure. Basically, interventions that target the source nodes are recovered, after which we can recursively extract the transitive closure of the graph. The authors also propose a variational autoencoder framework to estimate the latent causal representations from interventional datasets in practice. The standard Evidence Lower Bound is modified to include a discrepancy term, that roughly measures how well the interventional samples have been modeled. The model is then trained via the standard reparameterization trick. Experiments on a biological dataset, with around 100 latent variables, are performed. The model is shown to predict certain expected causal relationships as well as good R^2 values for unseen double-node interventions. The target audience are people interested in identifiable representation learning. The theory seems good but the assumptions are a bit strong and the experiments should have been more comprehensive. It's possible that similar experimental techniques will adapt for other datasets however it may require more work, since assumptions maybe limiting (see weaknesses below). #### References: - [1] A. Seigal, C. Squires, and C. Uhler. Linear causal disentanglement via interventions. arXiv preprint arXiv:2211.16467, 2022 - [2] K. Ahuja, Y. Wang, D. Mahajan, and Y. Bengio. Interventional causal representation learning. arXiv preprint arXiv:2209.11924, 2022. - [3] Varici, B., Acarturk, E., Shanmugam, K., Kumar, A., and Tajer, A. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.08230, 2023.

Strengths

- As opposed to the do-interventions that Ahuja et al. 2022 consider, soft interventions addressed in this work are more realistic. Similarly, unpaired datasets and unknown intervention targets are also more natural. - The illustrative examples shown in section 4.2 are useful to understand non-identifiability situations. - The intervention targets are learned on-the-fly, as part of the autoencoding variational bayes formulation. This allows the work to generate virtual counterfactual samples. This is shown also in the experimental section.

Weaknesses

- It's not mentioned anywhere in the introduction that the function f is a polynomial. Since this is a significant assumption both in theory and especially in practice (where we use neural networks), I find this very misleading to the readers. - Assumption 2 as it reads seems a bit strong. The authors show that certain quadratic SEMs satisfy this assumption in Appendix B, but do the more simpler case of linear SEMs satisfy them? If not, it's a bit strange that the theorem does not cover the case of linear SEMs with linear f, that's covered in [1]. - Image and genomics datasets are used as motivation, but it's not clear why the crucial assumptions such as interventional faithfulness or polynomial f, should be true in those settings. - Footnote 4 says that interventions are chosen to be shifts, why do the authors make this choice, when the theory seems to hold in general? This seems too restrictive and may limit potential applications. - Additional experiments on synthetic datasets would have been more illustrative of the technique's performance, as well as allowing access to ablation studies which are crucial for such a loss formulation. In other words, it's not clear if the good performance on the biological dataset is because of the model (as we would like) or instead just an artefact of the specific dataset and training technique used.

Questions

Some questions were raised above. - I think it's fair to also cite Varici, B., Acarturk, E., Shanmugam, K., Kumar, A., and Tajer, A. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.08230, 2023. - A minor remark, the problem studied in this work has also been called "causal representation learning" in the literature, while "causal disentanglement" is used for the special case when the latent variables are jointly independent.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Limitations have been discussed.

Summary

This paper focuses on latent causal representation learning, with a specific focus on showcasing the identifiability of the causal structure among latent variables. The authors propose an approach based on assuming a full-row rank polynomial generator, soft interventions on each latent variable, a generalized version of faithfulness, and sparsity, so that the causal structure can be identified up to some equivalence.

Strengths

Overall, the presentation is clear, and the study introduces a relatively novel contribution. While previous works in causal representation learning have primarily assumed marginally independent causal variables and hard interventions, this paper explores new dimensions by incorporating soft interventions.

Weaknesses

However, there is one primary concern regarding the interpretation of soft interventions. In line 111, the paper mentions, "We focus on the scenario where we have at least one intervention per latent node." On the other hand, Assumption 2 implies, "Intervention I with target i," which appears to indicate that only one variable is being intervened upon each time. To avoid any confusion, I kindly request the authors to clarify this point in the discussion.

Questions

In line 111, the paper mentions, "We focus on the scenario where we have at least one intervention per latent node." On the other hand, Assumption 2 implies, "Intervention I with target i," which appears to indicate that only one variable is being intervened upon each time. To avoid any confusion, I kindly request the authors to clarify this point in the discussion.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Response to minor points raised by the reviewer

We thank the reviewer again for the detailed review. Below, we provide responses to the minor points for completeness: > **”In Section 2, 3 and 4.1 ... not always clear which parts are novel ... which builds on previous results.”** Section 2 only introduced the model setup with citations to where the definition comes from. Papers in CRL that study similar setups are discussed in the previous Section 1.1 on related work. In Section 3, we laid out the definition of equivalence classes (for our main results), which many previous papers implicitly used but may not explicitly define. In Section 4.1, we stated its title as “Preliminaries”, so this is mainly built on previous work. In addition, when introducing the Assumptions and Lemmas, we stated the references in the paragraph above. To make this more clear, we will add the citation to the assumption themselves. > **"almost-linear mixing function and single-node interventions... should be clearly stated..." | "'unobserved causal variables confusing...”** We thank the reviewer for this suggestion. We will revise the introduction by adding the following sentence to the last paragraph where we introduced the content of this paper: “We provide theoretical guarantees when the latent variables are observed through a class of (potentially non-linear) polynomial mixing function proposed by [1].” We will also clarify the term 'unobserved causal variables' in line 52. > **”Example 1, Line 190: the interventional distribution of $U_2$ is presented as identical to its observational distribution. I assume this is a typo where the interventional distribution should be $P(U_2|U_1)=\mathcal{N}(U_1+1,1)$ instead?”** Yes, it should be $P(U_2|U_1)=\mathcal{N}(U_1+1,1)$ in line 190. Thank you for pointing this out; we will revise that accordingly. > **”Line 222: why do you consider here $p$ interventional settings ($I\in I_1,...,I_p$) instead of $K (I\in I_1,...,I_K)$ as in Sec. 2 before?”** This is for simplicity of illustrating the proof sketch of Theorem 1, as we stated in line 216. The formal proof in Appendix B.2 considers the general setting where there are $K\geq p$ interventions.

Response to Rebuttal

Thank you for your response and clarifications. The rebuttal clarified the theoretical contribution of the paper better, with the suggestion to also take these clarifications into account for the final paper version. The empirical part of the paper remains a weakness, e.g. given its several components necessary. Still, since I acknowledge the contribution to be mainly theoretical, I raise my score by one.

Opportunity to engage in discussions and answer further questions

Dear reviewers, we thank you again for taking the time to review our paper. We appreciate all the feedback and comments that help improve the work. As the discussion end date approaches, we would love to have an opportunity to engage in discussions, and provide further clarifications if there are additional questions. Once again, thank you for the considerations!

Response to rebuttal

I thank the authors for the response and clarifications. The additional ablation studies are quite important and the authors should add them to the paper, along with improving the writing as per the reviewers' suggestions. I'm willing to increase my score.

Response to author rebuttal

Thank you for the comprehensive discussion regarding the faithfulness assumptions, which I believe will significantly enhance the theoretical robustness of this paper. The authors have adequately addressed most of my questions, and I'm inclined to keep the score unchanged.