Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning

Summary

This paper proposes a novel contrastive self-supervised learning framework based on JEPA, namely C-JEPA. The main idea of C-JEPA is to address the limitations of the I-JEPA, especially the limited prevention of collapse with EMA, by incorporating the principles of VICReg. The authors demonstrate the effectiveness of C-JEPA through rigorous empirical and theoretical evaluations. They show that C-JEPA achieves superior performance metrics compared to existing frameworks with faster and better convergence.

Strengths

**[S1]** The paper is well-motivated that I-JEPA has room for improvement regarding the risk of model collapse and challenges in learning the mean of patch representations. They demonstrate that the proposed approach can effectively address the problem. **[S2]** The paper shows that the proposed approach consistently and significantly improves the performance of I-JEPA on various downstream tasks and shows the scalability of the approach. **[S3]** The overall writing is smooth and easy to follow.

Weaknesses

**[W1]** Though the paper shows a strong performance, I think the contrastive learning on JEPA lacks novelty. The proposed approach is a simple combination of I-JEPA and VICReg’s regularization strategy. **[W2]** Does C-JEPA perform better than I-JEPA with more pre-training, e.g., 800 or 1200 epochs? In figure 1, it seems that the slope of I-JEPA is much larger than C-JEPA. **[W3]** Invalid citation: Line 302.

Questions

Please address my concerns in the Weaknesses.

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

2

Limitations

They did not address the limitations and potential negative societal impact of their work in the paper. They must include the limitations in the final manuscript, e.g., the proposed approach requires a large computation and a large network capacity, therefore raising environmental concerns, e.g., carbon generation [1]. [1] Schwartz, Roy, et al. "Green ai." Communications of the ACM 63.12 (2020): 54-63.

Summary

The paper introduces C-JEPA, an enhancement to the Joint-Embedding Predictive Architecture incorporating Variance-Invariance-Covariance Regularization (VICReg) for non-contrastive self-supervised learning. This approach addresses limitations such as model collapse and inaccurate mean patch representations, enhancing stability and learning quality. C-JEPA shows improved performance across diverse tasks, including image classification, object detection, semantic segmentation, and video object segmentation.

Strengths

1. C-JEPA's innovative combination of VICReg with self-supervised learning architectures addresses critical issues like model collapse and enhances unsupervised visual representation learning. 2. The framework is supported by extensive empirical evidence and theoretical analysis, demonstrating superior performance compared with existing methods on multiple datasets and diverse tasks.

Weaknesses

1. The framework would benefit from comparisons with latest leading methods like Dinov2[1], Mocov3[2], and IWM[3] to benchmark against current advancements. 2. The paper claims that the inclusion of Variance and Covariance terms enhances training speed and stability (lines 291–295). However, the basis for these conclusions is not clearly articulated, necessitating further explanation to validate these claims. 3. While the paper presents results from pre-training on image dataset like ImageNet, its applicability to video-related tasks can be explored. An adaptation on video JEPA methods, such as V-JEPA[4] for video domains, could significantly extend the framework's utility. 4. There are typographical and formatting errors, such as the unexplained markers in line 302. [1] Oquab, Maxime, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023). [2] Xie, Zhenda, et al. self-supervised learning with swin transformers. arXiv preprint arXiv:2105.04553 (2021). [3] Garrido, Quentin, et.al. Learning and Leveraging World Models in Visual Representation Learning. arXiv:2403.00504 (2024) [4] Bardes, Adrien, et.al. Revisiting Feature Prediction for Learning Visual Representations from Video. arXiv:2404.08471 (2024)

Questions

Please refer to the weakness.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes

Summary

The paper presents C-JEPA, a novel framework integrating VICReg into the Image Joint-Embedding Predictive Architecture (I-JEPA) to address its limitations in preventing model collapse and learning mean patch representations. Empirical and theoretical evaluations demonstrate that C-JEPA enhances the stability and quality of visual representation learning, showing superior performance across multiple benchmarks.

Strengths

Rhe paper offers an innovative integration of VICReg with JEPA is innovative and addresses critical limitations in existing frameworks. Comprehensive validation i achieved through empirical evaluations and comparisons with state-of-the-art methods. The paper offers strong theoretical foundation supporting the benefits of the proposed method. The results support performance improvements in multiple tasks and benchmarks. Ablation studies and qualitative visualizations provide deeper insights into the method’s effectiveness.

Weaknesses

The additional regularization terms may lead to increased computational overhead. Further testing on larger and more diverse datasets is needed to confirm scalability and generalization. While the results are promising, testing across more varied domains would strengthen the conclusions.d The new combination of established SSL methods (I-JEPA and VICReg) comprises an incremental development in hte fiels.

Questions

Can you provide more details on the computational overhead introduced by the VICReg integration? Have you considered testing C-JEPA on more diverse datasets beyond ImageNet-1K and more diverse downstream tasks?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have acknowledged the potential computational complexity and the need for further testing on diverse datasets. They have made significant efforts to address the limitations of C-JEPA, and their proposed future work aims to explore these aspects further.