Summary
This paper proposes to adapt regularization components (variance, covariance) from the VICReg to supervised learning contexts, which aims to encourages the network to learn high-variance, low covariance representations, promoting learning more diverse features. The authors focus on an efficient implementation of VCReg and conduct experiments to show its effectiveness in transfer learning.
Weaknesses
1. **Lack of novelty and practicality.**
- Firstly, most SSL methods, such as VICReg, DINO and so on, have been verified to significantly outperform supervised learning on downstream transfer learning tasks. No matter what components are added, SL consistently focuses on matching label information. Therefore, in contemporary machine learning, SL is more commonly used for fine-tuning specific tasks rather than serving as a pretraining method for obtaining general-purpose representations. If the proposed SL+VCReg can outperform current self-supervised learning methods on downstream tasks, in my view, that would truly represent both novelty and practicality. I suggest that the authors conduct direct comparisons between their SL+VCReg method and leading SSL approaches on downstream transfer learning tasks. This would provide a clearer picture of where the proposed method stands in relation to current best practices.
- Secondly, the author proposes adding regularization components to multiple intermediate layers of the neural network. Although the runtime of each component is similar to that of batch normalization (BN), it still introduces additional computational overhead. Moreover, based on the experimental setup, selecting the coefficients $\alpha$ and $\beta$ for the two regularization components across different network architectures is challenging and requires significant cost. However, despite these additional burdens, the performance improvement is not particularly remarkable. These issues make it difficult for this method to have strong prospects in practical applications. The authors should discuss potential strategies for efficiently selecting the $\alpha$ and $\beta$ coefficients, such as automated hyperparameter optimization techniques. This would help readers better understand the practical implications of implementing VCReg.
2. **Some aspects of the method's setup are unclear and lack intuitive analysis.** Specifically, how VCReg uses spatial dimensions in its calculations is not well explained—I suggest that the authors provide explicit formulas. Additionally, it is unclear how the method determines which intermediate layers to add the regularization components. The lack of clarification on these critical settings raises concerns about the reproducibility of the method. The authors should provide a step-by-step algorithm or pseudocode for how VCReg is applied, including details on handling spatial dimensions and selecting intermediate layers. This would significantly enhance the reproducibility of the method and clarify its implementation.
3. **Some of the experimental results raise concerns.** CDNV measures the degree of clustering of the features, but the authors applied it to "two unlabeled sets of samples." It is unclear how the authors obtained and processed this data, and the significance of using unlabeled data in a supervised learning context is questionable. In Table 6, the results show that ConvNeXt (VCReg) demonstrates significantly weaker clustering ability compared to the standard ConvNeXt, which does not support the claim that this method prevents neural collapse. Furthermore, in Table 4, the reproduced performance of VICReg is even worse than SimCLR, which contradicts the prior knowledge of self-supervised learning. As a result, I find it difficult to be convinced by the experimental outcomes presented in the paper.