Lightweight Vision Transformer with Bidirectional Interaction

Summary

The paper proposes a new lightweight ViT structure called FAT. They use a fully adaptive self-attention mechanism for vision transformer to model the local and global information as well as the bidirectional interaction between them in context-aware ways. In addition, the paper introduces a fine-grained downsampling strategy to enhance the down-sampled self-attention mechanism for finer-grained global perception capability.

Strengths

- The paper is well-written and easy to understand. - The experimental results show the effectiveness of the method.

Weaknesses

- The usage of conv stem, conditional positional encoding, the convFFN and the aggregation of global and local attention is not novel. Thus, the novelty of this paper primarily lies in using bidirectional interaction. However, it is un-natural to explain via human visual system of using local to global and global to local interaction. Is it really the way human look at pictures? - In Eq.5, it is unclear why using different A1 and A2 makes the interaction 'bidirectional'. The author should further explain this. - Lack of comparison to the state-of-the-art models. For example, the proposed FAT-B3 has the same accuracy to CMT-S [1] but larger FLOPs and parameters. [1] CMT: Convolutional Neural Networks Meet Vision Transformers. CVPR 2022.

Questions

See weaknesses above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

See weaknesses above.

Summary

This paper presents a new family of lightweight vision transformers that are built on the Fully Adaptive Self-Attention module. The key innovation of these transformers lies in their bidirectional interactive module, which enhances both local and global features. The experiments primarily focus on image classification, object detection, and semantic segmentation tasks. Extensive evaluations were conducted on various common benchmarks, including ImageNet-1K, to assess the performance of the proposed transformers.

Strengths

The proposed method demonstrates a key innovation through bidirectional interaction between the global and local branches, leading to significant improvement as validated by comprehensive ablation studies. The study includes extensive experiments, and the ablation analysis is thorough, highlighting the quality of the research. The writing is clear and concise, effectively presenting the experimental setting and demonstration. Impressively, FAT-B0 and B1 achieve remarkable results in terms of both throughput and accuracy, providing valuable insights for future research on low-latency small models.

Weaknesses

1. The scaling behavior of the method is my primary concern. While I appreciate the author's focus, the current results indicate that larger models exhibit limited improvement over the baseline in the three tasks. In my opinion, it is essential to discuss the scaling aspect for a proposed general backbone. 2. The second issue is the lack of comparison in terms of inference time between different methods or different hardware (e.g. CPU/GPU). Based on the current results, the proposed model works very well under low parameter settings, providing the direct benefit of lighter and more efficient deployment on mobile devices. However, the author has not provided sufficient evidence to demonstrate whether the proposed method maintains efficient inference efficiency with such low computational requirements. This aspect is crucial for me in proposing a lightweight model because training speed is often not a concern for these types of models. Instead, focusing on comparing the capabilities of this model under different hardware and computational costs can significantly enhance the significance of this method. 3. The motivation for this paper appears somewhat forced, particularly from Figure 1. I have not come across any related work discussing this phenomenon. If any such works exist, please cite them, as it would enhance the trustworthiness of your motivation.

Questions

1. My first two questions are motivated by different considerations, and I recommend that the author concentrate on one aspect to augment the method's significance. Given the exceptional performance of the proposed model on small-scale counterparts, my particular interest lies in contrasting the inference across different hardware and CNN architectures. Nevertheless, should the author contend that the proposed model should possess broader applicability, I am also intrigued to observe its scaling behavior. 2. I am interested in investigating the training stability of the proposed methods. It would be helpful if there are any plots illustrating the training stability.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The author has discussed their limitation in the paper.

Summary

This paper presented an efficient vision transformer backbone for several tasks including classification, segmentation and detection. The key idea of this paper is on a new design of considering the interaction between local and global features. The experiments on imagenet, AD20K, and COCO have shown that the proposed method is comparable to most existing methods.

Strengths

This paper has clear contributions on the vision transformer design. 1. The proposed the local-global feature interaction is well motivated. 2. The proposed methods has been evaluated with comprehensive experiments. 3. This paper is well written and easy to understand.

Weaknesses

There are some weaknesses in this paper: 1. The area of new backbone is growing rapidly. Although the proposed method shows marginally improvements over the experimented baselines, compared with concurrent work, it becomes not appealing. 2. The proposed lightvision transformer is very hard to justify as it is not as efficient as mobile vit but also not sure how it can be scaled up. Thus it is very hard to justify. 3. The baseline methods are not consistent. For example Table 1 and Table 2 pick different methods for similar comparison. For example, ParC-Net should be put into 0~5 while the author intentionally put into different comparison setting to make the number look better.

Questions

The experiments are not fair comparison as some of the numbers are cherry picked. It will be more convincing to compare with Segnext, ParC-Net, MobileOne and mobileVIT clearly. As far as the reviewer can see, the proposed method is comparable or weaker than them. In addition, the ablation studies didn't show the bidirectional interaction is useful or not. For example, given single interaction will it be OK?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The key limitation of this paper is the proposed method lack of evidence on whether it can be mobile-friendly or scaled up as VIT liked backbones. Since the backbone is not general enough, it will has very limited usage.

Summary

This paper proposes a new family of light-weight vision transformers named FAT, which enhances the local and global feature fusion by a bi-directional interaction between them. Experiments on image classification, object detection, and semantic segmentation are conducted.

Strengths

1. The paper provides a good practice in designing high performance efficient ViTs, which could be a valuable reference for the future work. 2. The results on several benchmark datasets outperform all the compared models.

Weaknesses

1. An incremental work. There exists tons of works that focus on performing local and global features in ViTs (see Sec. 2 of the paper), this paper only proposes a simple way to enhance the feature fusion. Besides, other increments are actually a combination of existing successful practices. For example, the convolutional and non-overlapping downsample module is from PVTv2, the downsample in MHSA have also been widely used (e.g., PVT), CPE is from [1]. 2. Actually, a CVPR 2022 paper MixFormer [2] also proposes a bi-directional interaction of convolutional and self-attention branches. What's the pure difference between this method and MixFormer? 3. It is difficult for me to evaluate the efficacy of the proposed bi-directional interactions. The paper only compares the method with simple fusion baselines, while there exist many global-local feature fusion methods, I cannot figure out which one is better according to Table 1, since many increments are added in this method. I suggest the authors to replace the fusion method in FAT with other popular methods such as the interactions in MixFormer and message token in MSG-Transformer [3], for fair comparisons. --- **References** [1] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. In ICLR, 2023. [2] Chen, Q., Wu, Q., Wang, J., Hu, Q., Hu, T., Ding, E., Cheng, J. and Wang, J., 2022. Mixformer: Mixing features across windows and dimensions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5249-5259). [3] Fang, J., Xie, L., Wang, X., Zhang, X., Liu, W. and Tian, Q., 2022. Msg-transformer: Exchanging local spatial information by manipulating messenger tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12063-12072).

Questions

See Weaknesses.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes.

Reviewer-author discussion

Dear Reviewers, Please take a moment to read the authors' responses. Your insights and feedback are crucial in ensuring a comprehensive evaluation. Thanks. AC

Post review

I appreciate the authors for dealing with my concerns and including more experimental results. All of my questions are well resolved and this makes me more inclined to increase my score.

Thanks for the rebuttal

I appreciate the additional scaling experiments and the more detailed evaluation of inference efficiency. Most of my concerns have been addressed. While I still have reservations about the motivation, the results presented by the author demonstrate a commendable trade-off between efficiency and the model's performance, especially for downstream tasks. I have revised my scores upward.

feedback

Thanks for all the experiments. The additional results make me more convinced.