Summary
The paper introduces a knowledge distillation method for self-supervised learning on efficient network architectures such as MobileNetV3, EfficientNet-B0. The author proposes a simple technique called, RETRO, that reuses the teacher’s projection head for students during the knowledge distillation and this simple technique achieves significant improvements on self supervised learning results of efficient architectures.
Strengths
- The paper is easy to read and the method is quite simple and can be easily plugged into any self-supervised learning pipeline.
- The problem of improving the self-supervised learning for efficient architectures is important, especially on why these models are harder to train with SSL pretext task.
- Despite the simplicity of the proposed technique, RETRO achieves consistent improvements in all the tasks (linear evaluation as well object detection/segmentation) that the authors present and the results are impressive.
Weaknesses
- Despite its simplicity, the method seems merely an engineering trick which has worked on the knowledge distillation for efficient networks.
- Considering the limited novelty in the approach, I expected the authors would have put more effort in understanding why it works.
The paper also does not touch much on why efficient architectures are harder to perform SSL. Is there any specific reason based on the architectural design of these efficient networks? Generally, in supervised learning these efficient architectures are known to be better.
- Is this specific choice of keeping only the frozen projection head necessary in the student model? What if the consistency is maintained at earlier layers as well as the final embedding of the teacher and student networks. It should work in a similar manner.
Ideally, one would expect an ablation study on what part of the teacher network should be embedded in the student network to make SSL work?
- What if instead of the projection head, first few layers of the teacher network are enforced in the student model.
- Do the authors test the same approach of keeping projection head from SSL performed on one network to another (not necessarily efficient)? Can this save training time on another network architecture?
Questions
Please look at the weaknesses mentioned above.
Is the knowledge distillation in RETRO performed while doing pre-training on teacher model or after the SSL training has been performed on teacher model?
Rating
5: marginally below the acceptance threshold
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.