C-Disentanglement: Discovering Causally-Independent Generative Factors under an Inductive Bias of Confounder

Summary

The paper proposed Confounded-Disentanglement a method that makes use of inductive bias of a confounder by leveraging labels/knowledge from domain expertise. The aim is to recover the true/causal generative factors for the observed data which can recover a more meaningful latent representation that can be further useful for downstream tasks especially in the case of distribution shifts.

Strengths

- Originality: the authors propose a new way of leveraging observed confounders to learn disentangled representations. - Quality: the manuscript includes both theoretical and empirical evidence for the claims. - Clarity: I appreciate the schematics and toy running example to illustrate the intuition of the method. - Significance: Better latent representations useful for overcoming domain generalisation challenges is timely and important contribution.

Weaknesses

The problem of disentanglement, learning meaningful representations in general is an important , unresolved problem and I do agree that when additional information is available, we should make use of it, such as the scenario proposed by the authors of cdVAE. However, I do find the paper lacking in some recent relevant work and the overall presentation would benefit from including/positioning this method with regards to it. Please see more details in the question section.

Questions

1. A recent stream of disentanglment that also leverages additional/auxiliary information, similar to ICA but even closer to cdVAE are the following [1, 2, 3]. In those cases, the additional information is treated as intervention, or environment. Could you please position your work wrt that? What makes a confounder different than their auxiliary variable? 2. What are the advantages of interpreting the additional variable as confounder and not intervention? 3. There are other metrics for measuring the quality of disentangled representations that haven't been used R2, MCC for example 4. Have you tried the controlled generative process under your framework? What could you conclude from the results? [1] Lachapelle, Sébastien, et al. "Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA." Conference on Causal Learning and Reasoning. PMLR, 2022. [2] Khemakhem, Ilyes, et al. "Variational autoencoders and nonlinear ica: A unifying framework." International Conference on Artificial Intelligence and Statistics. PMLR, 2020. [3] Lu, Chaochao, et al. "Invariant causal representation learning for out-of-distribution generalization." International Conference on Learning Representations. 2021. ---- Update after Author response ---- I thank the authors for the detailed and timely responses, they have mostly addressed my concerns, so I increase my score to BA.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Yes.

Summary

This paper is motivated by the assumption that a few semantically significant generative factors generate real-world data. Then they study causal independence in generative methods by introducing a new framework, C-Disentanglement, that discusses the identifiability issue of generative factors regarding the inductive bias of confounder to handle the disadvantage brought by the statistical independence assumption of previous methods. At last, the authors implement their method, cdVAE, and evaluate it on three datasets. The results show that cdVAE performs better in the latent space compared to the existing methods and performs better in downstream tasks.

Strengths

The motivation is obvious, which emphasizes the importance of causal independence in generative methods and illustrates the advantages of causal independence for statistical independence, which is very interesting and important. To the best of our knowledge, the authors are the first to study how to explore causal independence confounders in generative methods. This article has complete content. It conducts a thorough investigation of the background. Then it proposes a new framework and elaborates in detail. Moreover, it has several experiments on three datasets to demonstrate its advantages.

Weaknesses

This article includes but is not limited to the following shortcomings: 1. There was much ambiguity about collaboration, which led me to rely a lot on my guesswork in understanding the article, like: a) D in line 141 and D in equation 7 do not seem like the same thing but use the same symbol. b) D_KL in Figure 4 and D in Equation 7 seem to represent KL divergence but in different symbols. c) What is the meaning of \Pi? I do not find its definition. Is that represent the soft assignment of samples? If it is, what is the connection between \Pi and \pi? 2. In Table 1, you evaluate your method on CelebA and Candle datasets. However, the measurement metrics between these datasets are different. You claim that the CelebA dataset does not have ground truth factors. However, the experiments on only two datasets cannot fully illustrate your methods' advantages, especially since CelebA does not have ground truth factors. 3. There are many kinds of generative methods, like the diffusion model. There is no comparison between your method and other generative methods. 4. The baseline vanilla VAE (2013), \beta-VAE (2016), FactorVAE (2018), CAUSAL-REP (2021), which do not compare, cVAE (2013), GMVAE (2016), are out-of-date.

Questions

1. I think implementing your ideas in VAE is a limitation for you. I hope you can implement your ideas in more generative methods. 2. The Related Work in the main paper is very similar to the Related Work in the appendix, so I doubt the significance of the Related Work in the appendix. However, I do not think this is a weakness of your paper, so I put this into Question.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

1 poor

Contribution

2 fair

Limitations

The authors point out the limitation of their method at the end of the paper; that is, the choice of C is difficult but essential, which is the same as our understanding of their method.

Summary

This paper delves into the domain of representation learning, focusing on the discovery of meaningful generative factors within a latent space. The authors propose the C-Disentanglement framework, which incorporates the inductive bias of confounders through labels or knowledge obtained from domain expertise. This framework enables the identification of generative factors even in the presence of confounding variables. The paper offers a comprehensive theoretical analysis and a practical approach to effectively identify these factors. Empirical results obtained from diverse settings validate the efficacy of the proposed method.

Strengths

 This paper is very well written and easy to follow.  The research addresses an important and fundamental question: how to identify Causally-Independent generative factors in the presence of common causes.  The idea proposed in this work is simple but certainly make sense. And the sufficient experiments validate the effectiveness of the proposed method.

Weaknesses

 The assumptions made in this paper are quite strict. One of the main concerns is that the paper assumes the probability distributions of the generative factors to be Gaussian distributions. This assumption may limit the applicability and generalizability of the proposed method to real-world scenarios where the underlying distributions may not strictly follow Gaussian patterns. Another strong assumption in this paper is the linear mixing of the generative factors. While this assumption might simplify the modeling process, it may not accurately capture the complex relationships and interactions between the generative factors in real-world scenarios. It is important for the authors to discuss the potential impact of this assumption on the performance and generalizability of the proposed method.  The proposed inductive bias in this paper is hard to obtain. The utilization of domain expertise knowledge as an inductive bias is an expensive and non-scalable approach.  The evaluations of the proposed method primarily focus on quantitative aspects, and incorporating qualitative evaluations would provide a more intuitive understanding of the method's performance.

Questions

 How can we ensure that the number of C is adequate?  In Line 215, Equation (10), the right-hand side of the equation involves several terms that are not enclosed within absolute value symbols. Does these terms are non-negative?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The authors adequately addressed the limitations of the work.

Summary

The paper proposes a new framework called Confounded-Disentanglement (C-Disentanglement) to learn disentangled representations by using inductive bias of confounding. Instead of the common statistically independent assumption for latent distribution which is usually not true in real-world datasets, the proposed framework partitions the observed data distribution based on a label set C and regulates the latent distribution separately for these subsets. The proposed framework is then implemented in a Variational AutoEncoder (VAE) setting, termed cdVAE. Practically, the proposed method models the latent distribution as a mixture of Gaussian model in which each individual Gaussian corresponds to a specific value of C. The paper shows the effectiveness of the proposed method on the image generation task on CelebA and Candle datasets, and on a classification task under distribution shift on 3DShapes dataset.

Strengths

- The paper extends research in disentangled representation learning by leveraging inductive bias of confounding that makes it more applicable to real-world datasets that contain correlations in generative factors. - The proposed framework and method are technically sound. - The results seem promising. - The proposed method is evaluated on multiple tasks and multiple datasets which shows the robustness of the proposed method.

Weaknesses

- The proposed method requires domain expertise knowledge to define the confounders. This can be easy in some cases but could be costly (in terms of time and cost) and error-prone in many cases. What would happen if an error is introduced in defining the label set? Also, how to distinguish from the cases where each subset of data (defined by a specific value of the label set) still contains correlations among generative factors e.g., in the example of Figure 3, if the imported apples contain large green apples and small yellow apples? Is it a requirement that the label set needs to be granular enough so that no correlations remain in the subset corresponding to each specific value of the label set? - What cases from section 3 do the experiments from the paper fall in? Please discuss this in detail. - The proposed method cdVAE effectively boils down to modeling the latent distribution as a mixture of Gaussians. How is it different from GMVAE [7] then? - Could the authors explain in more detail how the objective in CausalVAE [24] is fundamentally different from this paper? - The paper mentions that using a Mutual Information (MI) objective for latents is not sufficient/appropriate. Then, shouldn’t the evaluation metrics based on MI be avoided as well? Also, why not use DCI/D score in Table 2? - Can the authors explain in a bit of detail how all the evaluation metrics for disentanglement are being calculated and why UC and CG scores can't be calculated for CelebA? Can't the attribute annotations in CelebA be used as a proxy for generative factors? - The shape classification generalization experiment setting is a bit unclear. Can the authors please explain it again, maybe with an example? - All the results need error bars.

Questions

I have included all my questions that I would like the authors to answer in the weaknesses section above. The answers to all these issues are crucial to my final recommendation.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors have adequately addressed the limitations of the proposed method.