Summary
This paper is a courageous attempt at unifying understanding of most major self-supervised learning algorithms, across the two or three
families of them by now clearly identifiable. To this end, it seeks to explain non-collapse of the SSL objective by splitting it into two terms, akin to a cosine similarity term maximization with an additional term that can be seen either as a constraint or negative gradient.
Weaknesses
However, by now, it is a fairly well-known insight that SSL representations result from an equilibrium between the action of positive and negative terms (even if implicit), lest they collapse, and therefore that SSL losses can be split as the sum of two relevant terms [1, 2]. Hence the most critical issue in this paper is its treatment of prior work; for instance this perspective was readily known simply by generalizing contrastive methods away from log-sum-exp normalization and InfoNCE thanks to f-divergences [3] where the negative term comes from the Legendre dual representation of the objective optimized (this also generalizes easily to say SwAV). Other work have explained the negative term as coming for instance from architectural constraints [4] or from orthogonality through equivalence to gradient-based PCA [5,6,8] for non-contrastive.
Similarly the works both [7] and [8] position known SSL algorithms as instances of manifold learning and/or Riemannian SGD, which once again through its geometry explains the provenance of negative gradients. If one argues that unifying works can be assessed based on the generality of insights they provide or their empirical implications, then [7] nicely connects - or even proves equivalence of - known SSL algorithms to corresponding manifold learning algorithms; and [8] clearly explains how the stop-gradient operation is crucial in enforcing constraint in BYOL and SimSIAM, along with introducing multiple predictor-free variants based on that principle. This work's contribution unfortunately doesn't go as far in either direction as those two - and doesn't cite those published works either, thus not engaging with deep mathematical insights, or showing whether its empirical contributions are orthogonal or additive to those.
This is very relevant since observations in sections 2.2 and 2.3 have not only already been made elsewhere but also in more detail ('hϕ
learns to project za to zb almost perfectly' -> is detailed in for instance [8,9,10])... This lack of related literature research permeates all through the introduction, including the incorrect statement 'However, the exact collapse avoidance strategy of these [non-contrastive] methods is still unclear,' as it is in fact detailed and proven in [11, 8].
Two points of detail as well: first, the nested expectation minimization in the first term can also be optimized in a variety of ways, practically, and alternate maximization EM-style like in SimSIAM does not necessarily explain why a stop-gradient is necessary.
Second point of detail: the original EMA parameter in BYOL was 0.996 (with an annealing schedule all the way to 1); perhaps 'close to 1'
would be more accurate across all variants and datasets given what the authors aim to achieve. Also on the BYOL side, it is well known that weight decay is a critical factor in ensuring training stability [11,12] , so an analysis of that phenomenon rather than momentum in the optimization would be welcome.
Taken together, we feel that the contribution of this paper is not significant enough, and absolutely needs to be recontextualized using prior work. We would therefore encourage the authors to proceed to a rewrite furthering their interesting ideas, and consequently expand on the empirical results section - replicating algorithms from the below works as baseline on ImageNet-100 if compute is a concern.
[1] Wang & Isola, Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere.
[2] Tao et al, Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Framework.
[3] Zhang et al, f-Mutual Information Contrastive Learning.
[4] Liu et al, Bridging the gap from asymmetry tricks to decorrelation principles in non-contrastive self-supervised learning.
[5] Tian et al, Understanding Deep Contrastive Learning via Coordinate-wise Optimization
[6] Tang et al, Understanding Self-Predictive Learning for Reinforcement Learning.
[7] Balestriero & LeCun, Contrastive and noncontrastive self-supervised learning recover global and local spectral embedding methods.
[8] Richemond et al, The Edge of Orthogonality: A Simple View of What Makes BYOL Tick.
[9] Wang et al, Towards demystifying representation learning with non-contrastive self-supervision.
[10] Zhang et al, Align Representations with Base: A New Approach to Self-Supervised Learning
[11] Tian et al, Understanding Self-Supervised Learning Dynamics without Contrastive Pairs.
[12] Grill et al, Bootstrap Your Own Latent: A new Approach to Self-Supervised Learning.