Identifying General Mechanism Shifts in Linear Causal Representations

Summary

Recently, causal representation learning has drawn a lot of attention in the representation learning area, where it considers a causal relation among the latent generative factors. This work is under the setting of linear causal representation learning. Data $X$ comes from a linear mixing $X=GZ$ of the unknown latent factors $Z$, and latent factors $Z$ follow a linear SCM. They are trying to estimate the causal mechanism's shifting (causally changed) node, under the more general/relax intervention. They claim that ‘it is possible to identify the latent sources of distribution shifts while bypassing the estimation of the mixing function and the SCM (causal relation matrix). The experiments were provided to justify their setting.

Strengths

Pros: 1. This paper is very well-written. It's pretty clear and easy to understand. The reviewer enjoys reading it. 2. They are considered a more relaxed setting of intervention. It's an interesting idea where you directly find the shifted node.

Weaknesses

Cons: 1. The first issue is the main motivation of this work. Their idea is to bypass the estimation of the mixing function $G$ and the SCM $B$ (causal relation matrix). However, the reviewer believes this is the main motivation for causal representation learning. Why would we want a shift node, if we can't even identify the causal relationship in CRL? Could the author please clarify the point here? 2. The reviewer believes the strong assumption limits the significance of this work. The `access to a test function (Assumption B)' is a really strong assumption that seems not realistic. Since you got the permutation ambiguity fixed. And I believe you also fixed the sign ambiguity in your assumption. Combined with the bounded variance you basically got an ICA with only scaling ambiguity. What is the difference between this work and the classical linear ICA result? 3. The reviewer would like to see the clarification of the motivation and connection between linear ICA and this work. For linear ICA, we can get mixing function $M$ and the independent sources $\epsilon$ in an unsupervised way, with an identifiability guarantee. But for causal representation cases, the $M$ decomposes into $G$ and $B$. The natural/intuitive idea is to identify those factors with certain assumptions. Could the author provide any intuition for this? The reviewer would love to raise the score if these questions were resolved in the discussion session.

Questions

Besides the three questions I mentioned in the cons section, I do have another question here, but it does not lower the marks of this work, just discussion. 1. This question is not an issue for this work since it seems many reference works use interventional settings. But it would be interesting if the author could discuss this. Could the author clarify why we must use interventional data? Is there any alternative way since the interventional data is kind of a semi-supervised strong assumption? I know there is a paper about the 'Challenging Common Assumptions in unsupervised CRL', but is there any other way we can avoid using interventional data?

Rating

5

Confidence

5

Soundness

2

Presentation

4

Contribution

2

Limitations

N/A

Summary

This work studies the nontrivial problem of causal representation learning from the perspective of mechanism shifts within the latent SCM. Specifically, the authors relax existing restrictive assumptions in interventional causal representation learning, such as data generated from single-node perfect interventions and the number of environments necessary to facilitate identifiability and show that it is possible to identify the latent nodes that shift between environments/distributions from more general soft/hard and add/reverse interventions given access to fewer environments than the number of causal variables. Furthermore, the authors develop a practical algorithm to recover the latent sources attributable to the distribution shift and evaluate their method on synthetic data and a psychometric dataset. The main contribution in this work seems to be the use of a test function to score the noise factors learned from ICA to construct a sorted permutation matrix, which is used to construct the new unmixing matrix for scrambling the independent SCM noise variables.

Strengths

- The theoretical result and intuition of identifying latent sources of distribution shift is interesting and is a step toward more feasible CRL for real-world application. Most work focuses on a supervised discriminative setting for studying distribution shifts. This work stands out in being one of the first to identify latent sources of distribution shifts. - The empirical evaluation is extensive and considers real-world datasets to evaluate the proposed CRL algorithm. The results from the Psychometrics dataset suggest that the algorithm proposed is capable of identifying the latent shifts from the data distribution to a great degree and in line with human interpretation. The test statistic proposed to quantify the degree of distribution shift between nodes w.r.t unmixing matrix is practical. - Generalizing the class of interventions for interventional CRL to identify only shifted nodes is a useful result for real-world CRL, where interventions may be multi-node and more complex.

Weaknesses

- This work considers linearity in both the mixing and the SCM, which can be somewhat of a restricting assumption in practice. - Proposition 2 and Theorem 3 seem to contradict each other. By the current logic, the two statements imply that both shifted and non-shifted nodes require the same condition of the row corresponding to the variable index in the unmixing matrix to be invariant across environments (w/ sign flip). This should also be made clear in Section 4.2 in Step 3. - Minor points - In the caption of Figure 1, for the UK environment, it should be the edge Z_4 → Z_1 is removed instead of Z_5 → Z_1 removed.

Questions

- Proposition 2 and Theorem 3 are contradicting. What are the criteria for a node to be identified as a shifted node? Do you mean that if the ith row of the unmixing matrix is **different between environments**, then node i is a shifted node? This seems to be the case when looking at the proof of Proposition 2 in the appendix. I would appreciate it if the authors could provide some clarification on this. - Do the authors have any intuition about the setting where noise factors are correlated and the iid assumption of noise across environments is violated? In this scenario, we would no longer have the standard linear ICA result to build off of.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are discussed in the appendix.

Summary

This paper considers the setting of linear causal representation learning (CRL) with possibly multi-node interventions. Instead of focusing on the task of identifying the causal structure, which is recently shown to be impossible, the authors instead focus on the task of identifying mechanism shifts i.e. which nodes are intervened in each environment. The authors show that this identification task is actually possible, and design an identification algorithm to achieve it. Lastly, the authors empirically demonstrate the effectiveness of their approach.

Strengths

1. The paper is well-written and easy to follow. Most mathematical definitions and statements are supported with sufficient explanations. 2. The task that the paper focuses on i.e. identifying mechanism shift is quite interesting, and can possibly be considered in other cases where full identification is hard or even impossible.

Weaknesses

1. It seems that step 3 in Sec. 4.2 is stated without any proof of why it works. Is it just a heuristic or that it can provably lead to identification? 2. It seems to me that the main results of this paper is very closely related to [1], but the authors do not discuss in detail about this issue. (Please see the "Question part" for more details) [1] Jin, Jikai, and Vasilis Syrgkanis. "Learning causal representations from general environments: Identifiability and intrinsic ambiguity." arXiv preprint arXiv:2311.12267 (2023).

Questions

In the paper [1], the authors consider linear CRL (the same setting as this paper) and design an identification algorithm that works for general environments (i.e. multi-node interventions). Their algorithm can fully recover the causal graph, as well as recover the mixing matrix up to a surrounding node ambiguity (SNA). They show that SNA is am intrinsic barrier in this setting. What I'm wondering is that, given their identification algorithm, if it is the case that the task of "identifying mechanism shift" can be straightforwardly resolved. Because if you can recover the mixing matrix, then you can also recover the noise-to-latent matrix (i.e. $B^{(k)}$) in the current paper). Then you can identify the mechanism shifts simply by comparing the entries of different $B^{(k)}$'s. Of course, the mixing matrix is actually recovered with some ambuguities. However, given that such ambiguities are inevitable as shown in [1], I suspect that such ambiguities do not affect the task of identifying mechanism shift. *I am happy to raise my score if the above concern is appropriately addressed.*

Rating

3

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

N/A

Summary

This work studies the problem of detecting the mechanism shifts in a novel way by considering the latent nodes. The authors prove identifiability results based on assumptions softer from prior identifiability results for causal representation learning. Their method is based on ICA and is evaluated empirically on synthetic data and a real-world psychometric dataset.

Strengths

The paper is very well written and the scope and contribution are clearly stated and illustrated with examples. **Related work** The related work part is detailed and correctly positions the paper at the midpoint between causal representation learning and causal mechanism shift detection. **Novelty** The paper is novel as it proposes a methodology for a known problem (detection of causal mechanism shifts), in a new setting that considers latent variables, which is the case in the causal representation learning field. **Theory** The authors propose and theoretically prove a softer identifiability result that allows for fewer than $d$ and unrestricted interventions (soft/hard and possibly applied to multiple nodes). **Experiments** The method is evaluated on synthetic experiments and an interesting real-world experiment on a psychometric dataset, which provides evidence for its applicability in practice.

Weaknesses

This work has some possible weaknesses **Significance of contribution** The proposed work solves a simpler problem than causal representation learning, which doesn't require learning the whole mixing matrix $B$, but rather only the distribution shifts. I am concerned that this simplification of the problem might make it easier solvable and I wonder whether other methods of CRL could be transformed easily so they could perform well on this task. **Experiments** Following my previous concern, I wonder why you did not compare against prior CRL methods or methods for causal mechanism shifts. Such a comparison would enhance the experimental results. Can you adapt the CRL methods or the causal mechanism shift techniques for observable variables to your setting? **Sample complexity** From Figure 2 it seems that indeed for a large (towards infinity) number of samples, your method can accurately detect the mechanism shifts. I am concerned, however, that the sample complexity is high which makes nodes with over hundreds of nodes out of reach. This can also be seen from Table 1 where your performance drops quite early, even from graphs with 60 nodes. From a theoretical perspective, it would be interesting to compute how many samples are required regarding the number of nodes (sample complexity result - either theoretical or experimental study). Such a theorem would be appreciated by the community. **Real experiment: size of changes** The results are very interesting and agree with psychological findings. However, the number of nodes and changes are very small. A larger number of nodes (up to 100) would be more sufficient to show that your algorithm is valid in practice. **Limitations** You should explain either why the application to a large number of nodes is not needed in practice or add it to the limitations.

Questions

Line 56: Do you think that the problem you solve is relevant to identifying the locations of the root causes of a linear SEM (as in [1])? Figure 1 caption, 7th line: Typo, should be $Z_4\to Z_1$ Line 143: What are the implications of this assumption? Is it also present in prior identifiability results? Line 231: What is the effect of the observed space dimension $p$ being larger or smaller with respect to the problem you solve (does it become easier or harder)? Figure 3: the font in the legend and xticks must be larger. Line 293-294: Here I got a bit confused. This is different from what you show in the example of Fig. 1, right? Because in Fig. 1 you show interventions (shifts) across different countries. Line 297: Can you briefly explain how your methodology can be generalized to a nonlinear data-generating process? Line 303: What would distribution shift imply for image data? [1] Misiakos, P., Wendler, C., & Püschel, M. (2024). Learning DAGs from data with few root causes. Advances in Neural Information Processing Systems, 36.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have discussed some limitations in the appendix. However, a significant limitation of not being applicable to a large number of nodes is not included.

Thank you.

After your rebuttal, I am very positive that this is great work and I am accordingly increasing my score. Particularly I appreciate the novelty of your work, by combining two distinct problems (causal representation learning and causal mechanism shifts).

Response to the author

W1 The reviewer thanks the author for the clarification and agrees that it is of certain value for finding the shifted nodes. however, it does not fully resolve my question. The reviewer respectfully argues that the shift node setting is not aligned with the fundamental motivation of causal representation learning, which is finding the causal graph/mechanism. W2 The author answered my question about the strong assumption of the test function. The reviewer was satisfied with their explanation about relaxation. W3 For the motivation, the author referenced paper [1] is probably the closest linear model setting. It showed an interventional-based linear model. They showed the number of interventions needed for their method but that seems not a strict lower bound. The reviewer respectfully disagrees with the author's claim 'in the "inadequate" intervention data setting, estimating $G$ and $B$ from $M$ is impossible'. Please correct me if I was wrong, and I would also like to know if there is any reference proving the strict lower bound of intervention needed for identifiability.