Towards a Unified Framework of Contrastive Learning for Disentangled Representations

Summary

In this paper the authors propose a new dissimilarity measure function for contrastive representation learning. They provide the identifiability of the factors of variations in four different contrastive objectives. Specifically, they describe the connection between the learned representations and true factors using some function in the data generating process. They extend the property to cases where there are non-convex latent spaces or/and non-uniform marginal distributions. Empirically they validate their approach on several datasets.

Strengths

This paper is clearly written; The authors make it clear what the position of their approach in the literature of contrastive learning. The similarity measure function is simple to implement. I find it useful to discuss the strong and weak identifiability under different conditions, and the derivation is mathematically sound in my opinion. Though contrastive learning have been studies for a few years, it is interesting to investigate and unify the theoretical guarantees of disentanglement for a family of contrastive objectives under different assumptions.

Weaknesses

1. I am a bit confused about the problem definition. While the authors try to model the relationship between true factors and the learned representations, in most cases we don't really know the true factor. Instead what we typically do in contrastive learning is to measure the similarity between pairs of observations where there are slight differences from certain perspectives, i.e. one image is the rotated version of the other. So I would hope that the authors can talk a bit more about how the identifiability works in practical scenarios. 2. I wonder whether the authors need to any data augmentations when training with the contrastive loss. There does not seem to be any discussion on that.

Questions

Please see my questions in the previous section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

I think there is no potential negative societal impact of their work.

Summary

This paper presents an analysis focusing on the disentanglement of the characteristics of internal representations obtained by contrastive learning. The aim is to recover the internal representation of the true latent space up to some transformation by placing certain assumptions on the data generation process. Particularly, it generalizes Zimmerman's theoretical analysis so that it can be applied to some loss function variants and relaxes the data distribution assumptions. The paper successfully derives the results of identifiability without the common assumption of independence in the research of disentangled representation.

Strengths

While I could not fully follow the entire proof, the theoretical claim appears technically sound and solid. Including the analysis of the identifiability of Nonlinear ICA, the theoretical contributions obtained in this study are meaningful for the community focusing on acquiring disentangled representation with theoretical guarantees. Significantly, they have removed the independence assumption typically used in the analysis of disentangled representation. As it is difficult to sample all disentanglement factors in real-world datasets uniformly, this is an important direction for constructing practical theories. However, there are already several preceding studies on the approach of removing the independence assumption, so the positioning of this paper should be properly discussed, which I will mention in the Weakness section.

Weaknesses

**There are certain concerns about the gap between theoretical claims and realistic experimental settings.** The fundamental theoretical claims in this paper are based on the conditional distribution in the true latent variable space defined in Equation 1. As far as I understand the claims in the paper, the identifiability can be guaranteed by training with the contrastive learning objective by assuming that the positive pair samples can be obtained in the form following this latent conditional distribution. However, it is questionable how much validity can be claimed to assume that positive pair samples can be obtained in such a distribution form when considering practical situations. If the true joint distribution of positive pairs is considered known, supervised learning should be done by simply generating labels from the distribution, so the significance should be in the setting where samples can be implicitly obtained while the explicit form of distribution is unknown. This concern relates to the practical applicability of the theoretical claims concluded in the paper and is important in positioning the contributions and significance of this paper. For example, SimCLR, widely known for its practical effectiveness, uses two different views augmented from data as a positive pair, but can this be justified from the perspective of Equation 1? How meaningful is it to consider the corresponding latent variable space? **It is necessary to show the contributions of existing related studies appropriately.** While the paper claims to have successfully derived the results of identifiability without the typical independence assumption in the research of disentangled representation, there are already preceding studies that theoretically guarantee disentanglement without imposing independence assumption using weak supervision [1, 2]. In particular, while [1] adopts a GAN-based approach for training the model, it strongly relates to the problem setting of this study, as it uses pairs that share a part of the disentangled factors as supervision signals. What advantages can be recognized in the preceding studies compared to these approaches? **A fair comparison is needed between contrastive objectives.** In the paper, a slightly different objective function is adopted from the original spectral contrastive learning (SCL) objective in order to fit into the theoretical framework of the proposed method (Equation 5). Some numerical experiments have pointed out the numerical instability of SCL, but could this be due to the influence of modifying the original objective function? It should be verified whether the same problem occurs in the objective function presented in the original paper. If the numerical instability pointed out in this paper is due to the modification introduced, it should be explicitly mentioned, as it could potentially mislead readers about the effectiveness of SCL. **References** 1. Shu, R., Chen, Y., Kumar, A., Ermon, S., & Poole, B. (2019, September). Weakly Supervised Disentanglement with Guarantees. In *International Conference on Learning Representations*. 2. Locatello, F., Poole, B., Rätsch, G., Schölkopf, B., Bachem, O., & Tschannen, M. (2020, November). Weakly-supervised disentanglement without compromises. In *International Conference on Machine Learning* (pp. 6348-6359). PMLR.

Questions

- **In the numerical experiments using synthetic datasets, why were the values of $\beta$ not set to be compositional for each scenario?** For example, $\beta=1/2$ tested in the Box (simple) scenario does not seem to be tested in other scenarios like Hollow ball and Cube grid. Specifically, for the Cube grid scenario, only this condition seems to skip the intermediate value of $\beta=3$. Is there a clear reason for this? The validity of the parameter range used in the experiments should be clearly indicated. - **Sampling of positive pairs seems to be a crucial assumption in theory, but I could not glean from the text what assumptions were made.** In the Weakness section above, I have raise some concerns based on my understanding that positive pairs are sampled from a probability distribution derived from the conditional distribution of the true latent variables defined in Equation 1. Is this understanding correct? If this interpretation is wrong, it would be helpful to explicitly mention in the text about the assumption of the probability distribution of the positive pair samples.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

This paper analyzes the acquisition of disentangled representation using contrastive learning from the viewpoint of identifiability, successfully relaxing some of the assumptions required in previous research. This is a theoretically solid result, and the results have been verified in numerical experiments using several benchmark datasets. However, as mentioned at the beginning of the Weakness section, there are significant concerns about applying the claims of this paper to practical situations. By appropriately discussing the practical validity of the assumed premises, I believe the significance of this paper will become more firmly established.

Summary

This paper studies a unified framework of contrastive learning. The authors extend the theoretical result of [1] to embrace more contrastive losses, adapt it to marginal distributions beyond uniform distribution, and further relax some conditions that are required in [1]. The experimental results support the theoretical claim, and the authors also study/discuss the limitation of the theoretical framework. [1]: Contrastive Learning Inverts the Data Generating Process

Strengths

Overall, the story telling is good: The authors first come to a theoretical guarantees under some relaxed assumptions, then empirically validate the theoretical findings, and further analyze the impact of the violated assumptions, and discussed the limitations. Compare with previous close works, there are two advantages of this work, include: 1. A unified framework instead of focusing on InfoNCE family, and 2. More relaxed conditions of the theorems. In order to justify their theorem, the experiments on the synthetic datasets are designed to embrace more contrastive losses, and many scenarios are designed for non convex latent spaces, nonuniform marginals and without conditionally independent assumptions on latents. The studies on the limitations of the framework also bring some insights for future research.

Weaknesses

This paper has a lot of similarities with [1] (the 56th reference in the draft) in the foundation. Though this work does relax the theoretical guarantee of [1], the authors of [1] also empirically show that the underlying factors of variations can be identified even if the theoretical assumptions are severely violated. In Table 2 of this work, the authors also try to examine their theoretical framework under partially violated assumptions.

Questions

-- Regarding batch size, why the authors use 5120 in line 205, and 4096 in line 268? -- In this paper, for all the experiments, are users using the original SCL or the modified SCL (line 118-121). -- It is interesting to the authors conduct ablation studies on the number of latent factors, I have a few questions: 1. What scenario are you using for this experiment? 2. Compare with section 4.1, are you sampling more data? Or, as the dimension increases, are you also exponentially increase the data?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No negative impact found

Summary

This paper proves two identifiability theorems for a broad family of contrastive learning methods, without imposing the commonly used independence constraint. The theoretical findings are validated on several benchmark datasets, and the theoretical and practical limitations are fully discussed.

Strengths

1. The considered contrastive model family is broad enough to include several widely-used contrastive learning losses, in which some are never employed for disentanglement (SCL and NWJ objectives). This provides the paper with adequate novelty. 2. The proofs of the theorems are enlightening for me. In my opinion, the techniques used in the proofs will push this field forward. 3. The theoretical and practical limitations of the theory and the considered models are fully discussed, which is appreciated because the limitations of a work are of great meaning to readers.

Weaknesses

1. If the paper can discuss the physical or geometric picture of the conditions and theory, it will be much better for readers to understand the theoretical results. 2. The are some typos in the paper, e.g. Equation (23) on Page 4 of the Appendix misses a right bracket. 3. Some concepts are used in the paper but not introduced, like 'semi-metric' in Equation (1) and 'norm-induced metric' in Theorem 1. I will suggest the authors briefly introduce these mathematical concepts in the footnotes, which will enhance the readability of the paper.

Questions

1. why $\beta=2$ is excluded in the conditions of Theorem 2?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors have adequately addressed the limitations.

Summary

The paper develops a new theoretical framework for learning disentangled representations with contrastive methods. Building on the theory that contrastive methods approximately invert the data generating process, the paper extend it by proposing a new dissimilarity measure. The proposed framework supports a broader family of contrastive methods and relaxes the assumption about the data distribution and latent independence. Detailed theoretical proof is provided. Experiments validate the effectiveness of the proposed framework.

Strengths

Originality The paper is an extension of previous work [56] and does provide a new framework which can apply to more CL approaches and requires weaken assumption about both the data distribution and latent independence. I think the differences are obvious and the paper seems novel to me. Quality The paper is well supported by theoretical analysis. Codes are also provided. Clarity Overall the paper is well-organized.

Weaknesses

The author should provide more intuition and high level explanation of their designs (details in Questions).

Questions

- Could the author provide more intuition about the design of the dissimilarity measure (L106)? Are the optimal $\alpha$ and $\bar{\alpha}$ function unique? Why jointly optimize them with the network help? - For a more complex dataset in real task, what are the considerations when designing $\alpha$ and $\bar{\alpha}$ . Do you think a simple three-layer neural networks are also enough? Can you explain more on L287? What does "incorrect shapes" mean here? - It would be better if qualitative visualizations/comparison on more complex dataset (e.g. KITTI Masks or 3DIdent) are added to show the improvements.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

Authors propose a generalized contrastive learning framework extending the guarantees for disentanglement learning. The paper also links contrastive learning objective with identifiability results in both week and strong identifiability setting.

Strengths

- The paper is mostly well written and easy to follow (in few places it is hard to digest) - Authors do great job in converting all different contrastive learning objectives with unified notation - contrastive learning was empirically studied for it's effectiveness for disentanglement learning, authors to a great job in formally introducing these concepts

Weaknesses

- Some assumptions may be missing in case of thm1 (ref. questions for details) - Results on causal3DIndent are hard to interpret (ref. questions)

Questions

Here are some question that makes the paper bit unclear, addressing them can possibly improve the quality of the paper: Q1. To recover true latent (to establish identifiability) you need contrastive examples varying in only one particular generating factor, which might not be always possible in reality, are there any tricks to circumvent this issue Q2. In case of theorem 1, I believe there are some strict constraints on how \tilde{s} is constructed using s, these assumptions are not explicitly mentioned. (if there are no constraints, it's unclear how do you account for confounded generating factors) Q3. In case of confounded datasets, how would your identifiability result still hold? Since, in proof sketch you do not have any assumption over sufficiency and confounding factors. Q4. homeomorphism condition implicitly assumes the function to be bijective, continuous, and invertible which I believe is stronger assumption that [27] (correct me if I misunderstood the contribution). Q5. The experiments are not clearly detailed: how is the data generated? what are the data generating factors? Q6. why is R2 value >> MCC? : as higher R2 indicates higher correlation which should result in higher MCC, but opposite effect is observed in your experiments, can you provide justification for this behavior? Q7. More details on training procedure would be helpful (how is parameter b learned?) Q8. How does the bound on features affect correlation values? (based on table 4 results bounded MCC >> unbounded MCC, what could be possible reason for this? is it an artifact of clipping values?)

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Few technical limitations are mentioned in the paper, limitations of contrastive data acquisition in real-world setting is not discussed.

Response

Thank you for responding to my questions. **Gap between theoretical claims and realistic experimental settings** > We would also like to point out that there is a difference between knowing the distribution of observed pairs and latent pairs. The latter does not allow labels to be generated and is sometimes even partially known. This may motivate, for example, a partial labeling. I see that the authors and I agree on the differences between the former and the latter. While the partial labels mentioned by the authors may indeed be one of the practical assumptions, I still have slight concerns about the applicability of the authors' proposed problem setup as it is. In any case, I believe that the problem setting of this analysis should be clearly stated in the manuscript, specifically what practical situation is envisioned and the limits of its applicability. **Sampling positive pairs** Based on the authors' individual & general responses, I found that my initial understanding was correct. If so, my first concern still remains. I believe that we cannot say that the assumption that data can be sampled in IID from a probability distribution conditioned on a *true latent variable* is general enough (even if the latent variable is not observed). I think that if the scope of the application is properly narrowed down, this study would have enough significance, and the scope should be clearly mentioned in the introduction of the paper. Overall, I believe the theorems and the empirical results would be worth publishing, but their practical applicability should be clearly stated. Therefore, let me keep the original score.