Summary
The paper introduces the Guided Stop-Gradient (GSG) method that can be applied to SSL algorithms that adopt asymmetric dual encoders such as BYOL and SimSiam in order to boost their performance and stabilize their training. The idea of the GSG is to augment the loss function to attract different views of two different images instead of just one as it is done in BYOL and SimSiam. However, the method does not explicitly repel the representations of the views of the different images but carefully selects which positive views to attract with a stop-gradient operation. Experiments with pretraining are performed on BYOL and SImSiam models using ImageNet and CIFAR10 datasets and their trained model is applied on several downstream tasks.
Strengths
- The method is simple and practical to implement.
- Results are consistent and demonstrate increased performance on pretraining and downstream tasks.
- Using this method, models can be trained with lower batch size which is a plus.
Weaknesses
I believe that the paper in the current form lacks analysis (and intuition) that would help understanding why the method works better than the baselines. For example, I would like to see more analysis on the structure of the learned representation space, and the effect of the choice of the batch size and number of “negative” images added to the loss. Given that the experiments are performed on smaller datasets, this is feasible to add. See also my questions below.
Questions
I have the following questions:
1. I find that choices of k in Section 5.1. unintuitive. In the case of CIFAR10, wouldn’t a large k give more insights into how well the latent space is separated? On the other hand, using k=200 for ImageNet seems very large. Could you report the variation in labels for the retrieved neighbors, i.e., is the majority vote significant? This would be interesting to analyze also in case of CIFAR10 for larger k.
2. In linear probing, you train the classification layer for the same amount of epochs as the backbone and on a much larger batch size, which sounds a lot. Is there a reason for increasing the batch size to 4096? Have you tried with just a few iterations and smaller batch size?
3. I was wondering if you could add some discussion and analysis that would help better understand why GSG works well. It would be particularly interesting to understand why GSG isn’t affected by changes in the batch size. My hypothesis is that this is because the model isn’t explicitly penalized for misplacing (potentially many) negative examples. This is partially addressed in Appendix E, however, I find tSNE visualization subjective. Do you have any numbers supporting the separation of clusters, for example, std of representations of each class? Another experiment could be to add more images to your loss (x3, x4, …), would that hurt or boost the performance?
4. I am wondering what is the dimension of the representation space? Just to be clear, in my understanding you use z in all your experiments?
Minor:
5. In Section 5.2. it is not clear whether you finetune the backbone or use the frozen model.
6. When applying random stop-gradient in Section 6.1, do you randomly choose two terms out of 4 or do you randomly choose one term for each of the images? i.e, randomly choosing either first or second term and either third or fourth in Equation 2?
7. I believe that the claim in line 284 that the fluctuation of the accuracy in the beginning of the training of SimSiam is less severe when the model is trained with GSG is too strong without supporting numbers. Could you add some variance analysis?
8. Section 3 is helpful for understanding the idea of the model but it would be great if you could make it explicit that x1 and x2 are different images.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Code is provided but the paper checklist is not. Societal impacts nor limitations are not discussed.