FLSL: Feature-level Self-supervised Learning

Summary

This paper proposes Feature-level Self-supervised Learning (FLSL) to handle dense prediction downstream tasks. Specifically, the authors employ the transformer for joint embedding and clustering and construct the objectives from the mean-shift and k-means perspectives. Experiments show that FLSL yields significant improvements in dense prediction tasks.

Strengths

1. The paper is well-motivated and the proposed method seems to work well on dense prediction tasks. 2. The analysis of the connection between mean-shift clustering and SA sounds reasonable. The authors analyze the relationship between ViT and clustering from a new perspective. 3. Experimental results on detection and segmentation show significant improvements and demonstrate the effectiveness of FLSL.

Weaknesses

1. The relation between ADCLR and FLSL is not clear. I see both ADCLR and FLSL use cross-attention to learn patch-level information. So, what's the main difference between ADCLR and the inter-view objective? 2. ''feature-level'' looks somewhat misleading. For me, we typically divide the SSL into ''feature-wise'' (Barlow Twins [1], ZeroCL [2], ARB [3], VICReg [4]) and ''instance-wise'' methods (SimCLR [5], Moco [6]) by the objectives on the different dimension. The proposed method is more like a cluster-level method (SwAV [7]). [1] Zbontar J, Jing L, Misra I, et al. Barlow twins: Self-supervised learning via redundancy reduction[C]//International Conference on Machine Learning. PMLR, 2021: 12310-12320. [2] Zhang S, Zhu F, Yan J, et al. Zero-cl: Instance and feature decorrelation for negative-free symmetric contrastive learning[C]//International Conference on Learning Representations. 2021. [3] Zhang S, Qiu L, Zhu F, et al. Align representations with base: A new approach to self-supervised learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 16600-16609. [4] Bardes A, Ponce J, LeCun Y. Vicreg: Variance-invariance-covariance regularization for self-supervised learning[J]. arXiv preprint arXiv:2105.04906, 2021. [5] Chen T, Kornblith S, Norouzi M, et al. A simple framework for contrastive learning of visual representations[C]//International conference on machine learning. PMLR, 2020: 1597-1607. [6] He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 9729-9738. [7] Caron M, Misra I, Mairal J, et al. Unsupervised learning of visual features by contrasting cluster assignments[J]. Advances in neural information processing systems, 2020, 33: 9912-9924.

Questions

See weakness.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No clearly visible limitations.

Summary

This paper tackles self-supervised representation learning and primarily forces on learning representations for dense downstream prediction tasks, such as object detection and instance segmentation. To improve upon prior work, the main idea of the paper consists of two parts: (1) the paper leverages the underlying mean shift clustering process via the attention mechanisms of ViTs. In particular, the presented method relies on self-attention and cross-attention layers to achieve intra-level feature clustering. (2) In addition, the presented method takes a k-means perspective to achieve inter-level feature clustering.. The paper demonstrates the effectiveness of the approach via extensive experiments for dense prediction tasks, outperforming prior art for object detection and instance segmentation on MS-COCO.

Strengths

- The paper makes an interesting observation by connecting the attention mechanism (self-attention and cross-attention layers in particular) to mean shift clustering. This results in a relatively clean implementation (see figure 2 for an overview). - The overall idea is intuitive: 1. The feature representations that belong to a certain cluster are close to the cluster representative (center) and pushed away from the representatives of other clusters. 2. The cluster representatives of the positive areas are pulled together (positives) This is also reflected in the final loss function and can be implemented via a clean implementation. The pseudo-code in the supplementary materials is very insightful. - The presented approach demonstrates strong performance on multiple downstream tasks and datasets. MS-COCO is used for object detection and instance segmentation, UAVDT is used for object detection from UAV platforms, and DAVIS is used for video instance segmentation. The proposed method consistently outperforms DINO (a strong baseline) in these cases. - The paper contains ablations about the loss function and impact of the number of clusters K. The supplementary also includes additional details about the training setups. - Overall, the paper is well-written

Weaknesses

- A few comparisons or discussions with related works are missing. For instance, DeepCluster [a, b], PCL [c], and CDL [d] show similarities with the presented approach as they also leverage clustering. A comparison with CDL would be the most interesting as it relies on instance level and group level losses. - The presented approach is slightly more complex than prior work (DINO). In particular, the presented approach requires additional attention layers. As the architecture differs from conventional methods due to the introduction of the self-attention and cross-attention layers, what is the additional computational cost? This information is currently not present in the paper. - It’s also not clear how robust the method is towards certain dataset biases (e.g., object-centric datasets or imbalanced datasets). The approach relies on a uniform prior over the clusters, which is valid for the ImageNet dataset. However, prior works have been able to pretrain on uncurated datasets (e.g., SEER [e]). It’s currently not clear if this is also possible with the proposed method. It would be valuable to include experiments for COCO pretraining. [a] Caron et al., Deep clustering for unsupervised learning of visual features, ECCV 2018. [b] Caron et al., Unsupervised pre-training of image features on non-curated data, ICCV 2019. [c] Li et al., Prototypical contrastive learning of unsupervised representations, ICLR 2021. [d] Wang et al., Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination, CVPR 2021. [e] Goyal et al., Self-supervised Pretraining of Visual Features in the Wild, 2021.

Questions

How well does pretraining work on non-curated datasets, like the COCO or OpenImages datasets? What is the additional computational cost compared to DINO?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The paper briefly mentions the limitations near the end of the paper.

Summary

Current self-supervised learning (SSL) methods, including SimCLR, DINO, VICReg, and MOCOv3, focus mainly on instance-level representations, limiting their use in tasks like object detection and segmentation. To overcome this, a new two-level feature clustering SSL method named Feature-Level Self-supervised Learning (FLSL) has been introduced, which uses the mean-shift clustering process of Vision Transformers to improve semantic cluster representations. Experimentally, FLSL outperforms existing SSL methods in dense prediction tasks, delivering impressive results on multiple benchmarks, notably MS-COCO, UAVDT, and DAVIS 2017 datasets.

Strengths

1.It introduces the mean-shift and k-means for the pre-training of dense predictions. 2. The performance is improved with the proposed method.

Weaknesses

1. The idea of considering similar pixels or patches as positive pairs is not new. Many previous works have explored this idea already, e.g. [1]. 2. Many related works are not compared, e.g. [2, 3]. There are also some other works that I do not mention. [1] Hénaff O J, Koppula S, Alayrac J B, et al. Efficient visual pretraining with contrastive detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 10086-10096. [2] Xie Z, Lin Y, Zhang Z, et al. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 16684-16693. [3] Xiao T, Reed C J, Wang X, et al. Region similarity representation learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 10539-10548.

Questions

The authors claimed that they demonstrate for the first time the connection between the attention mechanism mean-shift clustering. However, there are already some works that demonstrate the connection between attention and clustering, e.g. [1]. Does this paper bring any new insights? [1] Zhou D, Yu Z, Xie E, et al. Understanding the robustness in vision transformers[C]//International Conference on Machine Learning. PMLR, 2022: 27378-27394.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have addressed the limitations.

Summary

The authors point out the limitations of previous SSL methods on dense prediction tasks, because of tis instance-level objectives. On the other hand, recent studies focusing on dense prediction based on region, patch, and pixel to learn globally semantic representations on these sub-regions. To this end, the paper introduces a novel method that learns representations with both local and global semantics by leveraging the mean-shift clustering. The proposed method consists of intra-view clustering with mean-shift and inter-view clustering with k-means. The extensive experimental results shows its effectiveness on various dense prediction tasks.

Strengths

* Well-written paper with superior performances * Conducted experiments on various dense prediction tasks

Weaknesses

* Some SSL methods that are strong in dense prediction tasks are omitted from the tables. (e.g., iBOT and RC-MAE) * For example, iBOT with ViT-S/16 outperform the methods in Table 1 in terms of AP^bbox and AP^mask. RC-MAE is also comparable to iBOT. * The algorithm table in the appendix seems to have a gap with the equations of the proposed method. Is there some omitted explanation in the algorithm table? * It would be better to compare the AAS of the method with other state-of-the-art methods so that the qualitative results can visualize distinctive behaviors. * Minor * It would be better for the highest values in the tables to be bold. [1] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. "ibot: Image bert pre-training with online tokenizer," ICLR'22 [2] Youngwan Lee, Jeffrey Willette, Jonghee Kim, Juho Lee, Sung Ju Hwang, "Exploring The Role of Mean Teachers in Self-supervised Masked Auto-Encoders," ICLR'23.

Questions

* Is the proposed method also competitive in terms of throughput? * The scales of random resized crop for each networks seem to have high lower bound compared to previous SSL methods. Is there any reason that the lower bound of scales should be relatively high? * Does the FLSL also show superiority in the fine-tuning task on ImageNet-1K?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The potential limitations are addressed well.

Summary

This paper presents FLSL a self-supervised learning model designed to give good performances of the pre-trained model on downstream dense prediction tasks. The proposed method can be summarised as a two-level clustering problem to achieve this objective : - Intra-view clustering: which aims to cluster together semantically related embeddings and pull apart those which are not. This is achieved using cluster representative which are obtained using mean-shift clustering. The embeddings are then drawn to their representatives. The paper demonstrated the relation between mean-shift and self-attention which is used of that purpose - Inter-view clustering: the goal is facilitate the representations of semantically similar features across the dataset to be close. This is done by applying a soft-assignment of the representatives obtained previously to a set centroids.

Strengths

1. The demonstration of the relation between mean-shift and self-attention is a strong contribution of this paper and it plays a key role in the intra-view clustering objective 2. The proposed method outperforms other SSL frameworks on standard dense benchmarks(ms-coco, uavdt, davis) 3. The paper is well written and the authors provided proofs for their claims as well as code and implementation details for reproducibility.

Weaknesses

1. This paper uses a bbox-aligned k-NN classifier to perform the ablation studies of their work, but we don't know how the performances on this classifier correlates with the results on downstream tasks segmentation or detection. 2. It would have been good to have further evaluations on semantic segmentation on other datasets such as ADE20k, PascalVOC, cityscapes to get a broader view of the performances of the model

Questions

1. what is the computational cost of the proposed method compared to other SSL frameworks? 2. Did you evaluate the importance of the different levels of clustering? like how much each participate to the actual benefits on downstream performances. I see some results in Table 5., but it shows the performance with k-NN and always considers the inter-view clustering. What happens when we do not consider the inter-view clustering? 3. Also noticed that using smaller patches on the backbone in FLSL (e.g. Vit-s/8) seems to be better compared to backbone with larger patches(16), do you know the reasons behind this behaviour? (in table 1 and 3)

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

no particular limitations which need to be addressed. The main limitations of this work can be the complexity of the method which the authors spoke about and the potentially computational cost of the framework.

Questions after rebuttal

I thank the authors for providing the rebuttal. I have 2 additional remarks: 1. As the approach takes ~20% longer during pretraining, I believe it makes sense to include 2 additional baselines: (1) increasing the number of layers in the backbone of DINO by ~20%; (2) increasing the training time of DINO by ~20%. 2. While I appreciate the images in the rebuttal, additional pretraining results on COCO would show that the presented approach can be applied to uncurated datasets (also see prior works mentioned in my original review). In addition, were the presented images randomly selected?

Official Comment by Reviewer taUM

Thank the authors for the detailed response. I tend to accept this submission.