Harnessing small projectors and multiple views for efficient vision pretraining

Summary

This paper investigates a theoretical formulation of the contrastive loss used in self-supervised pretrainging of vision models. (One of the assertions of the paper is that all SSL (self-supervised learning) losses are variants of the same loss). The authors propose a more compute-efficient version of the loss objective, and use the theoretic derivations of their analysis of gradient descent dynamics to propose two changes to SSL protocols in practice: the use of a higher orthogonalization regularization coupled with a lower-dimensional projection, and using more augmentations per sample. These recommendations are supported by experiments showing that a lower-dimensional projection coupled with a higher orthogonalization constant leads to higher top-1 accuracy results, and that using more augmentations per sample leads to higher downstream classifier accuracy and faster SSL convergence.

Strengths

Unfortunately the presentation in the paper was quite difficult to understand, which made it difficult to evaluate the paper's strong points (please see 'weaknesses' and 'questions' below. However, the following strengths still seem to be present. - There is a fairly strong coupling between the theoretical conclusions and the experimental results, making both more credible. - The use of a simplified (from a computational sense) loss objective to use a sample estimate (thm 3.1) is a helpful contribution that may help improving SSL training in general. - The experimental results are quite good in context of the baselines, showing higher accuracy at the same data size/similar accuracy at reduced data size, when the paper's recommendations are followed. - The authors make it clear where they are building on past results and where the novelty of their contribution lies.

Weaknesses

The paper is very difficult to read for several reasons: - Key terms are not defined inside the paper, making it necessary to reference multiple additional works in-depth just to get a basic understanding of this one. These include: * SimCLR * Barlow Twins * VicREG * backward data augmentation covariance kernels $k^{DAB}$ * Infinite Mercer Features and Mercer Theorem - Theorem 3.2 is only stated informally in the text, and is not restated formally in the appendix (though theorem B.2 seems like a restatement of the informal version of the theorem). Perhaps because of this, it is not clear at all from the statement of the theorem why SGD chooses redundant directions. - Typos around terminology, for instance in Figure 2, it seems that $\lambda$ should actually be $\beta$. Also, the scale for $\lambda$ in Panel B seems very different to that in Panel E. - The experiments use the new, proposed loss objective (derived from Theorem 3.1), which should agree with the more computationally-expensive ones in the limit. However, the discrepancy between using the two is not ablated, and no baselines are used that use the original loss formula, even where it might be computationally feasible (where $m$ is lower).

Questions

* What is the impact of using the new, proposed loss objective, as opposed to using the traditional one on the accuracies of the downstream models? * Why does theorem 3.2 imply the loss of orthogonality when using SGD? * What are the specific data augmentations used for the experiments?

Rating

6

Confidence

2

Soundness

2

Presentation

1

Contribution

3

Limitations

The limitations were adequately addressed.

Summary

The paper conducts a theoretical study the training dynamics of self-supervised learning (SSL) methods by using an idealized loss function that resembles the Barlow Twins/VICReg loss. The idealized loss function is built on the idea of forward data augmentation (\emph{DAF}) graph kernel. The paper shows how this loss function maybe recast in an equivalent form that is computationally friendly. The training dynamics behavior of this loss is then used to build a hypothesis that representation learning can be improved with stronger orthogonal constraint regularization for small dimensional projectors and multiple augmentations improve performance. The paper provides empirical evidence to support the claims made in the paper.

Strengths

- The analytical study conducted in the paper on training dynamics is very interesting to this reader and should be of interest to the community - The idea of using an equivalent loss to DAF that is simpler to compute may also be of interest to the community as it has connections to other recently proposed SSL methods - The practical takeaways may be relevant to SSL practitioners

Weaknesses

- A main weakness that I see is that I am unable to relate the faster convergence claim/proof to improved downstream accuracy. This may be a matter of presentation or perhaps a gap in the reviewer's knowledge on how alignment argument in pretext task implies better downstream performance. Would the authors be able to clear this question up during rebuttal? - The experimental setup considered in the paper is on the smaller-scale side for understandable reasons. However, this may make the statements made in the paper less interesting to empirical scientists that work in SSL. Experiments using ImageNet-100 may help with showing the claims can hold with setups used in practice. However, I do not see any experiments with ImageNet-100 in Section 4.1 (low dimensional projection)

Questions

- How would a reader understand claims related to downstream performance using your framework that argues for faster convergence with pretext task? - Do you have any results with Barlow Twins or VICReg using ImageNet-100 at 224x224 resolution images for Section 4.1? - The equivalent loss formulation calculates the mean of embeddings (in feature space). If my understanding is correct, the the following paper [Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations](https://openreview.net/forum?id=og9V7NgOrQ) could be of interest to the authors. Would the authors provide any insight on any possible connections in the paper? - The argument about lower dimensional projections is very interesting but appears to be opposite to empirical findings made by the following works: - [RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank`](https://openreview.net/forum?id=uGEBxC8dnEh) - [LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures](https://openreview.net/forum?id=f3g5XpL9Kb) Like above, are the arguments made in the paper consistent with the empirical findings made in prior works? - There appears to be a typo in Figure 2 where $\lambda$ is used instead of $\beta$. Please correct if applicable I look forward to getting clarifications during rebuttal for the questions above.

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have clearly listed their limitations. In addition, the empirical setup may not be the one used by empirical scientists in SSL so making a note may be helpful (but not necessary)

Summary

This paper identifies the implicit bias of non-contrastive SSL loss and optimization, and proposes two ingredients to improve SSL learning: 1) Low-dimensional projectors can yield good representations; 2) Multiple augmentations improve kernel approximation. Further, the authors propose that In a low-data regime, using diverse & multiple augmentations can be as effective as acquiring more unique samples. All of these insights get validated on experiments of ResNet-50-based models.

Strengths

1. The paper is theoretically motivated well based on the formulation of NC-SSL. By casting the problem into the equivalent formulation of data augmentation invariance loss and the orthogonality constraint. 2. The empirical evaluation fairly meets the expectations from the alternative formulation of NC-SSL, which validates the proposed insights.

Weaknesses

1. I am not sure about the definition of weak/strong orthogonality constraints. How did the authors define what is a strong or weak constraint? In other words, what value of beta is regarded as a strong constraint? 2. All the experiments are conducted on ResNet-50. Can authors do experiments on other architectures like Vision Transformers (ViTs) or other CNNs, in order to show the conclusions also transfer to other models? 3. The insights are empirically validated and reasonable. However, these are the things that a deep learning practitioner could expect or imagine, especially that data augmentation can help in the low-data regime and that multiple augmentation improves the performance. These insights are also consistent in the supervised learning setting. In other words, there is no big "surprise" from the paper; every conclusion can be predicted even without the knowledge of SSL. 4. For the conclusion that Low-dimensional projectors can yield good representations, the experiments are only conducted on CIFAR10 and STL-10, where the dimension size 8192 is considerably larger than the usual setting and even 1024 can be considered as a large projector. Can author did experiments on ImageNet-100? I am also curious the see the results on the step-wise increasing dimensions: 256,512,1024,2048,4096 instead of some big jumps in Table 1.

Questions

Please refer to the questions.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper is basically an investigation into the insights revealed by the alternative formulation of NC-SSL loss. The insights get empirically validated and the overall story makes sense. However, I do not see any "surprise" from the paper except for some predictable conclusions which generally hold in deep learning. I am also not really convinced by the results in Table 1, which has big jumps in dimension sizes and only validated on CIFAR10/STL-10. These issues prevent me from giving a higher score.

Summary

This work introduces a new loss for self-supervised retraining that is functionally equivalent to existing methods but is computationally more efficient. The method identifies a key equivalence between existing SSL objectives (VICReg and BarlowTwins) features when employing augmentation kernel. Furthermore, the authors demonstrate that under their new loss formulation far fewer projector dimensions are needed and in-turn the efficiency is improved (time to convergence). Additionally, the authors show that multiple augmentations improve the kernel approximation, improving accuracy and convergence speed.

Strengths

⁃ The paper is very well written and presented, with all contributions clearly defined and rationale within the problem setting. Figures are simple yet informative and lead to a high-quality piece of work that is understandable to a wide audience. ⁃ The proposed methodology is novel and highly original, addressing key and ongoing concerns in the field of self-supervised learning. Moreover, the method is theoretically and empirically supported with extensive proof and experimentation, thus addressing key concerns with correctness and confidence. ⁃ The authors present a variety of recommendations that go above simple proposition of a new methodology, thus contributing to the impact and significance of the work. ⁃ The empirical analysis is extensive and supports all claims made, the resulting performance is a faster convergence with fewer projector dimensions.

Weaknesses

I have only two minor concerns/weaknesses: ⁃ You enforce a orthogonalization constraint on the projector, yet how is collinearity of the features avoided in practice for earlier representations, specifically after the projector is dropped? Is this even a problem? ⁃ Alternative measures for efficiency could be presented alongside convergence speed. I would expect alternative measures in both iterations per second, and memory consumption to be included at the very least if making claims of improved efficiency. Additionally, how much of a speedup can be attributed to each component I.e. the reformulation vs orthogonality constraint vs reduced projector.

Questions

⁃ Does the proposed formulation lead to better improved overall performance, by that I mean improved best case downstream performance compared to original formulations? ⁃ Does number of layers or dimensions of intermediate layers of the projector have significant impact on the observed performance?

Rating

8

Confidence

3

Soundness

4

Presentation

4

Contribution

4

Limitations

The limitations are appropriately addressed

Response to Rebuttal

Many thanks for answering the questions raised during the rebuttal and addressing the identified weaknesses. I appreciate clarity on my misunderstanding on some points. I would emphasise that the inclusion of efficiently metrics and more clear comparative baselines on any revised manuscript. Given my original rating being positive, I for now am maintaining my score.

Response

> Understanding Downstream Performance Claims: The reviewer's question is important and touches on the core of representation learning in SSL. Our framework focuses on accelerating convergence of the SSL loss itself, which inherently influences downstream performance. By optimizing the SSL loss more efficiently, we ensure that semantically similar images are mapped closely in the representation space, facilitating easier classification through methods like k-nearest neighbors or linear decoding. This is why reducing the SSL loss more rapidly leads to more rapid improvements in downstream accuracy. Our empirical results confirm this (e.g. see Figure 3).We will emphasize this critical link in the revised manuscript to ensure clarity and robustness in our claims. - I thank the author for the above response. Figure 3 is does indeed show improved accuracy over time, however, I do not see the corresponding loss values during optimization. - In general, I understand the thinking above, but I do not believe smaller SSL loss always implies good representations (and better downstream accuracy). Trivial representations can provide zero loss but we want to avoid these situations. Empirically, recent SSL models (see I-JEPA especially the official logs (released by authors)[https://github.com/facebookresearch/ijepa] ) show a non-monotonic loss behavior which raises interesting questions on when a SSL user can conclude training is done. - As a SSL user, I have been curious about the question being considered by the authors. What are SSL losses really optimizing as training losses are always not informative. In any case, a careful argument in the paper by the authors using the rebuttal above would make the paper very useful to readers. I request the authors to make all other updates that they have committed to during the rebuttal. I am raising my score as I am satisfied with the rebuttal to my questions as well as reading other reviews and responses

Thanks for the response

Thanks for the response! My concerns have been addressed. Given my original score being positive, I would keep it as I do not see this paper presents an important contribution to the SSL community (the main theory paper is based on past results and conducts a empirical validation).

Thank you for the rebuttal

Thank you to the authors for the explanations and for committing to the improvements to the presentation of the paper. Accordingly, I am increasing my score.