DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions

Summary

The paper introduces a novel self-supervised pretext task for Vision Transformers (ViTs), called DropPos. It aims to enhance the spatial reasoning or location awareness of ViTs, based on the observation that ViTs are often insensitive to the order of input tokens. DropPos works by dropping a large random subset of positional embeddings, then using the model to predict the actual position of each patch based solely on its visual appearance. The paper identifies three major difficulties: (a) discrepancies between pre-training and fine-tuning (b) trivial solutions that fail to learn highly semantic representations by solving this simple task (c) patches with similar visual content. To prevent trivial solutions and increase task difficulty, this paper keeps only a subset of patches visible during the task. Given the potential similarity in visual appearances between different patches, the authors propose position smoothing and attentive reconstruction strategies. This relaxation allows for non-exact position reconstruction when exact positions are not critical. Quantitative evaluations demonstrate the effectiveness of DropPos, outperforming supervised pre-training and yielding competitive results against state-of-the-art self-supervised alternatives on various benchmarks.

Strengths

1. The main claim is attractive. It would be quite interesting (and a bit counter-intuition) to see the vision transformers can learn a very good representation by such a simple patch predicting task, which is very coarse-grained. Particularly, the results of this paper look competitive to the mainstream pre-training tasks. 2. This work provides extensive ablation studies to verify their design and to reveal some insights. 3. It proposes Position smoothing and Attentive reconstruction to solve the problems like patches may share similar visual appearance.

Weaknesses

1. The paper requires additional experiments and deeper analysis to substantiate some assertions. For example, this paper starts from "Vision Transformers (ViTs) are quite insensitive to the order of input tokens, the need for an appropriate self-supervised pretext task that enhances the location awareness of ViTs is becoming evident.". Although this claim fits my intuition, it is weird that after citing some papers [13, 39, 60], the paper does not talk about this claim any more in the following sections. What would happen if messing up the order of input tokens? Or even further, should this really be viewed as a drawback? Insensitiveness to the order may also be a good property. The authors touch on this in the related work, but this claim needs more attention since it's the main drive of the paper. Also, it's unclear if DropPos helps this issue. More specifically, if a model is pre-trained by DropPos and then finetuned in ImageNet, would it be more sensitive to the order of input tokens? Looking into this would help us understand if the improved performance comes from the model being more "sensitive to the order of input tokens". 2. A recent work: Jigsaw-ViT: Learning jigsaw puzzles in vision transformer. Another work Jigsaw-ViT proposes to include solving Jigsaw in the training of ViT, which is close to the task of this work. It is beneficial to include this for comparison. [1] Chen, Yingyi, Xi Shen, Yahui Liu, Qinghua Tao, and Johan AK Suykens. "Jigsaw-ViT: Learning jigsaw puzzles in vision transformer." Pattern Recognition Letters 166 (2023): 53-60.

Questions

I am not really sure so I put this in Questions instead of Weaknesses. After checking the main paper and supplementary, it seems the authors did not mention if the pre-training is conducted with DropPos only, or together with other pre-training tasks like in MAE. Could the authors answer whether DropPos is the only training objective? If the answer is yes, the reviewer would hope the authors can discuss a bit about why such a very coarse-grained task can be more powerful than the counterparts. Does it mean the dense visual cues are not important in pre-training? Overall I am feeling this paper is a good trial, although it may need more in-depth analysis and appropriate discussion about relevant works.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper introduces a novel approach to self-supervised representation learning for vision transformers, focusing on enhancing their positional awareness. The authors proposed a new pretext task called DropPos, which involves reconstructing the positions of dropped tokens in partial observations. By leveraging DropPos, which is a spiritual adaptation of MAE, the authors consistently achieve superior results compared to the state-of-the-art (SOTA) techniques in standard downstream tasks, including image classification, object detection, and semantic segmentation.

Strengths

The paper effectively addresses the problem of incorporating inductive bias into vision transformers, and it takes an interesting approach by employing self-supervised learning (SSL) techniques. Enhancing the positional awareness of vision transformers is a significant aspect, as it can greatly improve their performance in downstream tasks that rely on location awareness and positional information. The proposed pretext task, DropPos, is simple yet effective, which aligns with the requirements of SSL. Additionally, such pretext tasks are efficient as they involve masking a substantial portion of the input data. The paper highlights that using tasks like DropPos eliminates the need for careful selection of target representation and mask strategy, as typically performed in mask image modeling. Furthermore, the paper is well-written and easy to follow.

Weaknesses

One potential weakness of the paper lies in its experimental evaluation. While the proposed pretext task is commended for its simplicity and effectiveness, it would have been valuable to compare the performance of a DropPos pretrained Vision Transformer (ViT) with a hierarchical transformer such as Swin, pretrained using mask image modeling [A, B]. This comparison would have provided insights into whether a carefully designed ViT architecture already addresses the need for positional awareness, rendering the pretext task redundant. Furthermore, the paper lacks an in-depth analysis of what the ViT is actually learning and how it achieves positional awareness. Quantifying the extent of positional-aware representation learning is crucial. Analysis such as intertoken distance within a layer, sparsity of attention weights, and linear probe results are missing, which would have shed light on the underlying reasons for the success of the proposed pretext task. A significant concern arises in the evaluation section where the authors reproduce most of the state-of-the-art (SOTA) results reported in other papers. However, the reproduced numbers are generally lower than the original reported results, without any mention of the differences. It would be important for the authors to investigate the causes of these discrepancies, considering factors such as differences in the experimental setup (e.g., number of GPUs, changes in batch size and learning schedules) or potential missing engineering tricks. Particularly concerning is the lack of significant improvement (wrt MAE) in object detection and segmentation results, which are crucial for spatial modeling. Lastly, the authors missed referencing relevant literature on contrastive learning approaches designed for spatial modeling, such as [C]. In fact, [C] outperforms DropPos when trained for 200 epochs on the ADE20k dataset. [A] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai and Han Hu. SimMIM: A Simple Framework for Masked Image Modeling [B] Xiang Li, Wenhai Wang, Lingfeng Yang, Jian Yang. Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Locality. [C] Patch-level Representation Learning for Self-supervised Vision Transformers, CVPR 2022

Questions

1. Why do we need to compute Affinity as indicated in equation (9). Why can't we use the self-attention matrix (softmax(KQ'))? 2. In Table 3, it will be also interesting to check the results of 2->1. 3. What happens when ViT is pretrained with DropPos for longer epochs like 1600 epochs? 4. In fig 3, how do we know that which predictions are correct? For rebuttal, please also refer to the points mentioned in the weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The paper acknowledges a limitation in its conclusion, which the reviewer agrees with. Although the experiments conducted with ViT-B are deemed sufficient to demonstrate the potential of the proposed method, it is crucial to provide a comprehensive explanation and understanding of the model, regardless of its size. This ensures a robust analysis and comprehension of the proposed approach beyond the specific ViT-B architecture.

Summary

This paper presents a simple yet effective approach for generative self-supervised representation learning on images, namely DropPos. The proposed approach drops a large random subset of positional embeddings for visible tokens and classifies the actual position for these tokens via visual appearance. Experimentally, DropPos outperforms state-of-the-art self-supervised approaches on a wide range of downstream benchmarks including image classification, detection and segmentation.

Strengths

- The paper is well written and organized. - The idea in this paper is simple yet effective, which brings something new in generative self-supervised representation learning. - The authors provide clear implementation details (e.g., Pseudo-Code), which makes it easier to be reproduced. - Detailed ablation study is conducted to verify the effectiveness of the proposed approach. - Achieving SOTA performance on various downstream tasks.

Weaknesses

- As illustrated in (4), the cross-entropy loss is applied for dropped position supervision. Except for this loss, is additional loss used? e.g., the MSE reconstruction loss used in MAE. - In MAE, more layers are used in the decoder (i.e., 8). Here the decoder only consists of 2 layers. Is it because the dropped position classification task is easier than the patch reconstruction task? - What’s the overall training time for DropPos? Is it comparable or more efficient than existing self-supervised approaches? - For completeness, please add more dropout-guided work for discussion, e.g., DropMAE: Masked Autoencoders with Spatial-Attention Dropout for Tracking Tasks (CVPR23), which similarly employs dropout mechanism, but for spatial-attention dropout in videos.

Questions

No. See Weaknesses.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

No.

Summary

This paper introduces DropPos, a self-supervised pretext task designed to enhance the location awareness of Vision Transformers (ViTs). By dropping positional embeddings and reconstructing the positions of visible patches with some auxiliary strategies, DropPos improves spatial reasoning abilities in ViTs. Experimental results demonstrate the efficacy of DropPos, outperforming supervised pre-training and achieving competitive performance against state-of-the-art self-supervised methods. The paper provides good insights on enhancing location awareness in ViTs for future works.

Strengths

1. The motivation is clear. The paper addresses the motivation of enhancing positional awareness and spatial reasoning abilities in vision transformers for pre-training. 2. The method is simple. The DropPos focuses on reconstructing dropped positions with some heuristics to avoid trivial solutions and ambiguities. Compare with the contrastive learning, it does not need complicated augmentations. 3. The method is effective. Compared to contrastive learning or masked image modeling, the proposed DropPos exhibits faster pretraining and slightly improved performance on several benchmarks.

Weaknesses

1. The initialization of the positional encoding for the DropPos is not discussed, and there is no analysis of the impact of the different PE initialization strategies for the proposed DropPos method. 2. The experiments are insufficient to validate the motivation. Although preliminary evidence is provided by experiments on downstream tasks such as detection and segmentation, which indicates that the DropPos enhances the location-awareness of ViT, the paper lacks more in-depth and intuitive experimental analysis and discussion to verify the strengthening of position sensitivity of ViT by the DropPos method. 3. The description of the DropPos method is not specific and clear enough, including the method implementation, the flowchat of pseudo-code in section 3.2. 4. The scaling properties of the DropPos on ViT are not explored in depth. For example, compared with the MAE method, It is questionable whether the method remains effective on Vit-Huge compared to the MAE. Experiments on more advanced vision transformers, such as Swin-Transformer, are also encouraged to be conducted. 5. Compared with the MAE method, there is a lack of studies on transfer learning using iNaturalists and Places, and the robustness evaluation on ImageNet.

Questions

1. Regarding the DropPos method: Could you provide more specific details and clarity on the implementation of the DropPos method and the flowchart of the pseudo-code in Section 3.2? This would help readers better understand the proposed approach. 2. Strengthening of the validation of motivation. Could you enhance the experimental analysis and discussion to provide more depth and intuition regarding the impact of DropPos on the position sensitivity of ViT? Can you consider conducting additional analyses or visualizations to better illustrate and explain the observed effects? 3. Positional Encoding Initialization: It would be valuable to discuss the initialization strategy for positional encoding in the DropPos method. How was it initialized, and were different initialization strategies explored? Analyzing the impact of different positional encoding initialization strategies on the performance of DropPos would enhance the comprehensiveness of the study. 4. Scaling Properties and Generalization: Could you further investigate and discuss the scaling properties of the DropPos method on larger architectures, such as ViT-Huge? Is the method equally effective and robust on larger-scale models? Additionally, have you considered applying the DropPos method to other advanced vision transformers like Swin-Transformer and evaluating its performance? 5. Transfer Learning and Robustness Evaluation: It is noted that there is a lack of studies on transfer learning using iNaturalists and Places datasets, as well as the robustness evaluation on ImageNet. I would like to see if the DropPos method remains effective on these benchmarks. 6. The Fig.~3 about the qualitative results of position reconstruction is somewhat confusing and requires a clearer explanation.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

N/A

Summary

This paper a method for self-supervised representation learning. Given a ViT architecture, the authors propose to predict the absolute position of masked positional embedings at random. Although the general direction is not new, the authors pose it in a simple an interesting way, that achieves good performance in the downstream tasks.

Strengths

1) The proposed ides is simple yet effective, and is executed well. 2) The presented is presented properly and is easy to follow. 3) The authors evaluate their method properly on several downstream tasks, achieving acceptable performance.

Weaknesses

The direction of obtaining a supervision signal from the patch positions in the ViT architecture has been explored before. The authors mention the majority of them in their related work section but do not discuss what the advantage of their proposed method is. The performance supports the effectiveness of their method, but they could discuss more explicitly their advantages and ablate on that. Moreover, a comparison with [1] would be informative. [1] Sameni et al., Representation Learning by Detecting Incorrect Location Embeddings, In AAAI 2022.

Questions

Please see above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, they have.

Reproduced numbers

The reviewer acknowledges and appreciates the authors' efforts in addressing all raised concerns. It's acceptable to not include the model results from 1600 epochs. The incorporation of the Swin results, coupled with the in-depth analysis, will undoubtedly enhance the quality of the paper. The reviewer accepts the model configuration for the COCO and ADE experiments as stated: "For efficient training, we perform 12 epochs of fine-tuning instead of 100 epochs on COCO, and 80k iterations instead of 160k iterations on ADE20k, following [48] and [18]." However, there seems to be a discrepancy between the reported figures in this paper and those presented in [48] and [18]. This discrepancy has not been explained. For instance, the BootMAE results appear to have been under-reported. Additionally, it should be noted that the MAE results (trained for 1600 epochs) in the BootMAE paper outperform those of Droppos.