Understanding the Role of Equivariance in Self-supervised Learning

Summary

This paper provides a theoretical understanding of equivariant self-supervised learning (E-SSL) methods and their effectiveness. The authors propose an information-theoretic analysis that explains the synergy effect between class information and equivariant transformations, leading to improved downstream performance. They also identify three principles for E-SSL design: lossy transformations, class relevance, and shortcut pruning. The paper further discusses the importance of model equivariance and demonstrates its advantages in E-SSL. Overall, the contributions of this paper lie in the theoretical explanation of E-SSL and the guidance it provides for designing effective E-SSL methods.

Strengths

1. The article is well-structured, and the arguments are clear. 2. The author analyzes how equivariance affects self-supervised learning from an information theory perspective and proposes three principles for designing E-SSL. 3. The theoretical validity is demonstrated through theoretical proofs and some small experiments.

Weaknesses

1. The experiments are conducted on few datasets which is not very convincing. 2. There exist some typos in the paper, such as the “xIn” in line 352. 3. It is better for the authors to provide a design enhancement example and conduct experiments to prove the effectiveness of the example, as it only explains the success of previous E-SSL works.

Questions

1. Is 89.49% mentioned in line 279 the ACC of the SimCLR framework during testing? Even with the enhancement of SimCLR, the task of predicting rotation only achieved a maximum effect of 59.06 in downstream tasks. These two values are not very close. 2. In the paragraph starting from line 328, it is believed that the action space of the instance discrimination task is N, but I think it is actually a combination of many binary classification tasks with an action space of 2. So I think contrastive learning is not a special case to equivariant tasks. 3. Principle 2 states that it is best to be category related, does it mean that category label information needs to be known during pre-training stage in SSL? 4. Why can predictive patches like MAE be considered as an equivariant SSL task?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes.

Summary

This paper aims to reduce the gap between theory and practice for equivariant SSL, which refers here to the sub-family of discriminative SSL methods that do not enforce the invariance of representations to augmentations. The authors propose an explanation based on the "explain-away" (_“Explaining away” is a common pattern of reasoning in which the confirmation of one cause of an observed or believed event reduces the need to invoke alternative causes._) and conclude that this principle justifies strictly positive mutual information between the pre-task labels (e.g., the rotation angle) and the downstream task label (i.e., semantic class label) thereby explaining why the pretext and downstream tasks are not systematically misaligned.

Strengths

- Relevance of the research question: while an intuition as to why E-SSL performs better than a random encoder might be easy to formulate (see below in weaknesses), a theoretical formulation supporting these intuitions eludes. A principled explanation would help the community in the design of better and more principles E-SSL approaches.

Weaknesses

My main concern regards the soundness of the contribution, the reasoning leading to this concern follows: - in this work, the authors use the "explaining-away" principle which relies on the proposed latent variable model shown in Figure 2 to justify E-SSL performance. Authors consider a partitioning of the latent information in $C$ and $S$, the former encapsulates semantic information, the latter the non-semantic information (should hence include the object position & orientation, textures). - I believe a reasonable intuition as to why E-SSL (e.g., predicting the rotation angle of an image as a pretext task and then performing semantic classification) can perform better than a random encoder - as shown by authors in Figure 1 -, is because of a selection bias in images used for training: each semantic class/object is more likely to be depicted in a specific orientation (e.g., a person standing) than in any other orientation (i.e., a person upside down). This means that the pretext and downstream tasks are not completely misaligned: being aware of the class information allows one to make a better guess about the rotation angle than a random guess. If an object is as likely to be depicted in one orientation than any other orientation than the class label is uninformative to make a guess regarding the rotation angle. The difference in results between horizontal and vertical flip in Figure 1 supports that intuition. The explanation proposed by the authors does not support that intuitive explanation of E-SSL but instead seems to propose a overly-simplified - The explaining-away principle justify a strictly positive mutual information between $\bar{X}$ and $A$ which authors extend to a strictly positive mutual information between $\bar{C}$ and $A$. It is not obvious from the get-go how authors can theoretically justify this extension. To summarize, the two aforementioned points should be further discussed by the authors to confirm the soundness of the proposed explanation which seems to take shortcuts both from an intuitive and theoretical perspective. I would suggest discussing these points more thoroughly. Minor comments: - Relevance of certain results: Theorem 2 is tied to an over-simplified mixing model ($ X = A + C$) - Rigor: - Notation: notations should be consistent throughout the paper (e.g., line 250 and 256, line 236/237 and line 240) and assumptions should be made clear (e.g., eq (3) and line 293, lambda=1 is not stated) - Principle III: refers to theoretical results - "style features may also explain the equivariant target A" - , which have not been explicitly shown (in section 4.1 or in the appendix) - Contrastive learning vs. Instance discrimination: while instance discrimination performs classification with a number of classes equivalent to the number of training instances, contrastive learning like SimCLR usually discriminates at a batch level hence I believe line 339 does not accurately describe contrastive learning but rather instance discrimination. - References for claims: "style features are often easier for NN learning" - line 260, - Typos: line 352, 68

Questions

- Can you please clarify what is meant by "global" vs. "local" transformations (principle II)? - Can authors rigorously justify the extension of the explaining-away principle from X to C?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Limitations are not explicitly discussed but do mention the use of simplified models (e.g., section 4.3) for theoretical purposes.

Summary

This study contributes theoretical insights that enhance our understanding of conventional practices in SSL training. Building on these theoretical foundations and supported by experimental evidence, the study puts forward several principles for the practical implementation of equivariant self-supervised learning designs.

Strengths

This study presents theoretical findings corroborated by experimental results. Specifically, the theory enhances our understanding of the design of augmentation functions. These findings are crucial as previous designs of augmentation functions have largely relied on empirical exploration.

Weaknesses

Experiments are conducted using smaller-scale models. Typo in line 352.

Questions

Could the principles 1,2,3 be applied to other domains beyond vision-based SSL?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Please refer to the question.

Summary

This paper proposes a theoretical and empirical study of the role of invariant and equivariant representations in self-supervised learning. While a number of works have been focused on learning equivariant representations, it remains unclear whether or not equivariance is beneficial in specific tasks. By studying how applying/predicting transformations in latent space is a task that requires class information (through the explaining away effect) the authors derive insights on the behaviour of existing methods and provide directions for future work, notably that invariance may not be necessary at all to obtain good representations.

Strengths

The theoretical analysis confirms previous empirical findings, for example that the considered augmentations must be complex/lossy [1] as well as diverse [1] to improve how much class information is present in the representations. The collider structure and explaining away phenomenon are an elegant way to understand the interplay between the original data and its transformations, showing that C and A cannot be considered to be independent. While invariance loss are commonly used to learn equivariant representations[2,3] (in conjunction with an equivariant objective), previous work had found that even with a purely predictive objective, competitive performance can be obtained[1,4]. This is in line with the conjecture line 281 "learning from equivariance alone can achieve competitive performance", and raises an important future line of work. [1]Garrido, Quentin, et al. "Learning and leveraging world models in visual representation learning." arXiv preprint arXiv:2403.00504 (2024). [2] Gupta, Sharut, et al. "Structuring representation geometry with rotationally equivariant contrastive learning." arXiv preprint arXiv:2306.13924 (2023). [3]Devillers, Alexandre, and Mathieu Lefort. "Equimod: An equivariance module to improve self-supervised learning." arXiv preprint arXiv:2211.01244 (2022). [4] Garrido, Quentin, Laurent Najman, and Yann Lecun. "Self-supervised learning of split invariant equivariant representations." arXiv preprint arXiv:2302.10283 (2023).

Weaknesses

The paper seems to focus on two distinct families of methods. Methods which are augmentation aware (e.g. RotNet) and methods which are truly equivariant, which are explicitly designed to learn the transformation between transformed data in latent space, bringing more structured representations. While this distinction may not affect the general ideas and theoretical arguments, it may impact empirical results and practical considerations. It would be good to have a discussion/comparison of these distinct approaches. Most of the practical insights stemming from this work are qualitative. A conclusion such as "knowing C is beneficial to predicting A" is a useful general guideline, but doesn't directly translate to practical guidance. The results would be stronger with more quantitative assessments of class relevance (perhaps in another, more computationally friendly domain), i.e. estimating I(A;C|X). This should be seen more as an avenue for future work though.

Questions

In experiments such as figure 1, what is the training loss/method considered ? Theorem 1 puts forward an argument which seems key to the success or not of equivariant SSL (when used without an invariant objective), "How much information about C do you need to know to predict A". Bringing this point (and class relevance in general) forward in section 3.1 would make the motivation from figure 1 clearer.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Limitations are addressed in section 6, focusing on the theoretical nature of the work and how it doesn't directly translate into improved methods. It would be good to elaborate further on the limitation of the analysis, such as how different instantiations of an equivariant object may lead to different behaviors, or the difficulty of characterizing quantities such as I(A;C|X) for various transformations.

Answer to Rebuttal

Thank you to the authors for their efforts in addressing my comments. I have re-read the paper and read the other reviews/rebuttals. I am happy to increase for the following reasons: - authors have cleared my concerns regarding the soundness of their theoretical results (i.e., use of explaining away effect) - going through the material once again, I see better how my intuition can be explained by the author's work, I believe connecting the author's contributions with a higher level intuition the reader might have regarding E-SSL and why it works would benefit the paper and hope the authors will adjust the manuscript accordingly. Overall, I believe the paper contributes to a better understanding of E-SSL methods. I believe the manuscript should be updated to incorporate the distinction between CL and ID made by the authors in the rebuttal for completeness.