SparseFormer: Sparse Visual Recognition via Limited Latent Tokens

Response

__W1. Comparison with token pruning__ Thank you for your valuable suggestion. We here compare SparseFormers with one of the most typical and effective token pruning method, ToMe [1], which can be used off-the-shelf to trained models. The throughput is measured on a single NVIDIA A5000 with batch size 32. Note that AugReg [2] models here are pre-trained on ImageNet-21K and we use them as strong baselines. | model | acc | throughput | FLOPs | |:--------------:|:----:|:----------:|--------------| | DeiT-S | 79.8 | 977 | 4.6G | | DeiT-S, ToMe@13 | 78.5 | 1606 | 2.4G | | DeiT-B | 81.8 | 305 | 17.5G | | DeiT-B, ToMe@13 | 80.0 | 481 | 10.4G | |--------------|----|----------|--------------| | IN-21K pretrained AugReg-S | 81.3 | 977 | 4.6G | | IN-21K pretrained AugReg-S, ToMe@5 | 81.0 | 1065 | 3.7G | | IN-21K pretrained AugReg-S, ToMe@13 | 79.3 | 1606 | 2.4G | | IN-21K pretrained AugReg-B | 84.6 | 305 | 17.5G | | IN-21K pretrained AugReg-B, ToMe@13 | 82.5 | 481 | 10.4G | |--------------|----|----------|--------------| | SparseFormer-T | 81.0 | 1207 | 2.0G | | SparseFormer-S | 82.0 | 824 | 3.8G | | SparseFormer-B | 82.6 | 475 | 7.8G | The table shows that even with ImageNet-21K pre-training, the post-training ToMe with AugReg-B only just matches the same performance as SparseFormer-B, which was trained from scratch on ImageNet-1K. The smaller SparseFormer-T also offers a better trade-off over AugReg-S w/ ToMe. __W2. Why the detection results are not more favorable__ This could be because we did not include any form of positional information in the latent token embeddings, as described in Section 4 model configurations. As a result, the attention between these token embeddings is not aware of their positional relationships. Possessing positional information is crucial for achieving optimal detection performance, as validated in the original DETR [3]. The detection transformer pipeline relies on positional information for accurate localization and to suppress redundant detections by utilizing attention between tokens. Meanwhile, the segmentation is simple in sense that we can perform 'classification' on latent tokens that correspond to specific areas in an image, and map this 'classification' back into nearby pixels in the original image space using a simple operator (the location-aware cross-attention we used). Besides, using the exact same SparseFormer for the classification and its weights might be a reason, and we expect that introducing more newly initialized transformer blocks could alleviate this gap.

Response (cont'd)

__W3. Ablation study on the RoI adjusting mechanism__ We have included the ablation study on the RoI adjuting mechanism from two aspects in Table 3 (b) and Table 3 (c) in the submission paper. We repeat these two tables here. | method | SF | ViT/32 | ViT/32* | conv×4 | swin | |-----------|------|--------|---------|--------|------| | top-1 acc | 81.0 | 72.8 | 74.3 | 79.4 | 79.7 | | GFLOPs | 2.0 | 1.4 | 1.7 | 2.2 | 2.0 | Table 3 (b) shows the effectiveness of the focusing transformer to adjust RoIs and extract features to build 49 tokens, compared with the dense counterparts to produce 49 tokens. All entries in the table use the same cortex transformer configuration. The 'ViT/32' produces 49 tokens simply by patchifying an image into $32\times32$ patches, rather than using the focusing transformer. The 'ViT/32*' add two more transformer blocks to 'ViT/32'. The 'conv×4' uses 4 convolution+relu after the early convolution, where each convolution is with stride 2 and doubled output channel to produce 49 tokens. The 'swin' exploit the shifted local attention used in [4] to produce 49 tokens, similar to the 'conv×4'. The focusing transformer shows its importance to adjust RoIs to prioritize foregrounds and exclude backgrounds, and therefore enables SparseFormer to perform more accurate recognition with fewer latent tokens, compared with dense counterparts to produce tokens. | $L_f$ | top-1 | GFLOPs | |-------|-------|--------| | nil | 77.8 | 1.6 | | 1 | 79.7 | 1.7 | | 4 | 81.0 | 2.0 | | 8 | 81.0 | 2.5 | Table 3 (c) studies the number of iteration of the focusing transformer to adjust RoIs, $L_f$. The 'nil' stands for the token RoIs are just learnable parameters of the model and do not adapt to different image content. In other words, the RoI for a token does not adjust at all and keeps the same for all images in the 'nil' entry, and as expected, it is with an inferior result. We can see that the iteration number is vital to the final performance, as the focusing transformer with the insufficient iteration may not effectively adjust tokens to foregrounds. [1] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. ICLR 2023. [2] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Trans. Mach. Learn. Res., 2022, [3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ECCV, 2020. [4] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. CVPR, 2021

Response

Thank you for your reviewing our paper. We will address your concerns in the following: __W1. Similarity to Perceiver and DeformableDETR architectures__ First, we would like to emphasize again that the aim of SparseFormer is to build a _sparse_ architecture for visual recognition tasks, and many efforts of this paper are put on _how to perform visual recognition with much fewer tokens in a transformer-like architecture_. This distinguishes SparseFormer from either Perceiver or DeformableDETR, where both latters do not attempt to reduce the computation. __Difference with Perceivers__ We agree with that the primary distinction lies in the focusing transformer since its introduction enables SparseFormer to perform accurate recognition with a highly limited number of tokens. From Table 3(b) in the paper, where illustrate how 49 tokens are produced, one can see the effectiveness of the focusing transformer comparing with dense counterparts. As noted in the paper, Perceivers use a naive cross attention layer to traverse the pixel or feature space in a dense manner $\mathcal{O}(H\cdot W\cdot C)$, where the focusing transformer in our SparseFormer extract a few tokens from an image in a sparse way $\mathcal{O}(N\cdot P\cdot C)$, regardless of the input height and width. Also, the size of latent space is also sparse, the typical Perceiver use 512 tokens with 1024 dimension, resulting in a total 512\*1024 capacity, which is more than $8\times$ the SparseFormer-B uses (81 tokens with 768 dimension). __Difference with DeformableDETR__ Building a sparse architecture necessitates a sparse variant of the basic operators on the image space. We must admit that DeformableDETR and SparseFormer share the similar idea to use bilinear interpolation to extract image features, but the motivation and details differs a lot: DeformableDETR aims to achieve the shift invariance in detection transformers (DETRs), where the naive cross and self attention layers in DETRs struggle to handle, while SparseFormer uses bilinear interpolation to efficiently sample image features. Also, deformable attention in DeformableDETR is typically performed on _the highly semantic feature map (e.g., after ResNets)_ and _in a multi-scale manner_, while SparseFormer uses _a single-level early conved feature map_ to perform sampling _in the very beginning of the network_. From this view, SparseFormer can be seen as a _decoder-only_ architecture for visual modeling except for the early convolution, which much differs from DeformableDETR that requires deformable self attention encoders on dense units upon the backbone. __W2. Dense prediction tasks and information loss__ SparseFormer can perform dense prediction tasks like object detection and semantic segmentation. We have included experimental results and discuss them in Appendix A.1 and Appendix A.2. Specifically, Table 7 in the Appendix shows the detection results on MS COCO on a simple SparseFormer-S with only an extra linear classifier head and an extra MLP for regression, and the training recipe and loss also following the DETR. Given that no positional information is injected into latent tokens and no multi-scale features are used, SparseFormer can still achieve meaningful detection results with about $0.3\times$ FLOPs. Table 8 in the Appendix illustrates the segmentation performance of SparseFormer on ADE20K. One can see that SparseFormer reaches the result on par with the dense Swin-T w/ UperNet with $0.18\times$ GFLOPs. We deeply understand your concern about information loss introduced by SparseFormer since the latent token capacity (e.g., 81\*768) is indeed much smaller than the image size (e.g., 3\*224\*224). However, we believe that this is _not_ a significant issue for visual recognition tasks. Natural images contain a lot of pixel-level redundancy, such as backgrounds, which contributes little the final recognition. In fact, our sparse paradigm is exactly based on this redundancy and the proposed SparseFormer learns how to discard the redudancy, or how to perform 'information loss', with the recognition supervision. As for generation tasks like with diffusion models, this information loss issue becomes somewhat critical, and it is appealing to investigate how to incorporate SparseFormers into generation models with the least information loss. However, this is beyond the scope of this paper.