FasterViT: Fast Vision Transformers with Hierarchical Attention

Response to Reviewer oF4Y

We sincerely thank the reviewer for their time and efforts in reviewing our work and providing valuable feedback that can further strengthen our manuscript. Below please find our detailed responses: > **Differences between HAT and attention block in EdgeViT** There are key differences between our proposed HAT and the attention block in EdgeViT. The attention in EdgeViT follows a “local-global-local” paradigm. Specifically, a convolutional layer first aggregates local information into representative tokens. A global (sparse) attention is then computed on the aggregated tokens. Lastly, a depthwise convolutional layer is used to compute local information from global information. In contrast, HAT computes self-attention between: (1) each carrier token and their corresponding local regions (2) all carrier tokens. The carrier tokens are only computed once and propagated to next layers, whereas EdgeViT repeats the computation for each operation at every transformer block. One disadvantage of the proposed scheme in EdgeViT is the loss of information in transitioning from one attention block to another, which is sub-optimal. Another potential drawback is that the way information is propagated is restricted by convolution operations; meanwhile, HAT allows one to learn the aggregation and propagation mechanics via attention. > **Impact of conv-blocks on throughput** Thank you for your comment. We have conducted an additional ablation study to demonstrate the effect of conv-based block on both accuracy and throughput as shown below. | Model |Top-1 | Throughput |-------------------|----------------|----------------| | FasterViT-0 |82.1|5802| | FasterViT-0 wo Conv-block |81.7|3616| | FasterViT-1|83.2|4188| | FasterViT-1 wo Conv-block |82.8|3280| | FasterViT-2 |84.2|3161| | FasterViT-2 wo Conv-block |83.8|2085| | FasterViT-3 |84.9|1780| | FasterViT-3 wo Conv-block |84.5|1397| | FasterViT-4 |85.4|849| | FasterViT-4 wo Conv-block |84.9|712| According to our experiments, replacing Conv-based blocks with Transformer-based counterparts significantly reduces the throughput while also reducing the accuracy. As expected, the Conv-based blocks are more efficient than the transformer counterparts for processing larger input sizes. The model with conv-based blocks also has higher accuracy compared to their fully-transformer-based counterparts due to incorporating inductive biases such as locality. The combination of Conv-based (stage 1 and 2) and transformer-based (stage 3 and 4) architecture as presented in FasterViT strikes the right balance between accuracy and efficiency. We have added this ablation study to the revised manuscript. > **Throughput comparison on other platforms** We have added additional throughput comparison on different platforms such as V100, TITAN RTX and A6000 GPUs, Jetson Nano and Intel(R) Xeon(R) E5-2698 v4 CPU. For all comparisons, we show that FasterViT achieves a Pareto-front in for ImageNet Top-1 and throughput trade-off, hence validating the effectiveness and scalability of our model to different hardware platforms. In V100 ([Figure](https://drive.google.com/file/d/1FP0pDnZMdxw2Sy0z1tX-8A54T6eKD0cq/view?usp=sharing)) or TITAN RTX ([Figure](https://drive.google.com/file/d/1elnLkDMM5s2ch-E5MKb5YDj1vV_Rsx1s/view?usp=sharing)) comparisons for instance, we observe a strong performance for all FasterViT models. Even on CPU ([Figure](https://drive.google.com/file/d/1iVsev1XxgNaFEgFrhV0i4lGO-Fldl3z9/view?usp=sharing)), FasterViT still outperforms competitors by a significant margin. Please see the supplementary materials of the revised manuscript for all the figures regarding throughput and Top-1 accuracy.

Response to Reviewer Wvdn

We sincerely thank the reviewer for their time and efforts in reviewing our work and providing valuable feedback that can further strengthen our manuscript. Below please find our detailed responses: > **Differences between proposed attention and local approaches like Swin** Specifically, Swin uses a local-only window-based attention and window shifting. However, window shifting is sub-optimal for modeling cross-region interactions due to the limited stride. On the contrary, in our work, we use carrier tokens to model cross-region interactions (global attention) as well as interaction between the carrier token and local regions. Hence, this allows for effectively modeling both short and long range dependencies in a very efficient manner. > **Minor improvements in terms of accuracy while Reduced throughput when using HAT** Thank you for your comment. We believe that accuracy improvements in Table 5 are significant when compared to other models. Specifically, for FasterViT-1, using HAT improves the performance by 1.1% in terms of Top-1 accuracy. For comparison, a mere 1.1% improvement in terms of ImageNet Top-1 accuracy equates to scaling from ConvNeXt-T (82.1%) to ConvNeXt-S (83.2%) models. Furthermore, the drop in the throughput is expected since HAT effectively models the cross-region interactions that may not be properly captured by Twins and EdgeViT. > **Throughput comparison on other platforms** We have added additional throughput comparison on different platforms such as V100, TITAN RTX and A6000 GPUs, Jetson Nano and Intel(R) Xeon(R) E5-2698 v4 CPU. For all comparisons, we show that FasterViT achieves a Pareto-front in for ImageNet Top-1 and throughput trade-off, hence validating the effectiveness and scalability of our model to different hardware platforms. In V100 ([Figure](https://drive.google.com/file/d/1FP0pDnZMdxw2Sy0z1tX-8A54T6eKD0cq/view?usp=sharing)) or TITAN RTX ([Figure](https://drive.google.com/file/d/1elnLkDMM5s2ch-E5MKb5YDj1vV_Rsx1s/view?usp=sharing)) comparisons for instance, we observe a strong performance for all FasterViT models. Even on CPU ([Figure](https://drive.google.com/file/d/1iVsev1XxgNaFEgFrhV0i4lGO-Fldl3z9/view?usp=sharing)), FasterViT still outperforms competitors by a significant margin. Please see the supplementary materials of the revised manuscript for all the figures regarding throughput and Top-1 accuracy.

Response to Reviewer tsUS

We sincerely thank the reviewer for their time and efforts in reviewing our work and providing valuable feedback that can further strengthen our manuscript. Below please find our detailed responses: > **Throughput comparison on other platforms** We have added additional throughput comparison on different platforms such as V100, TITAN RTX and A6000 GPUs, Jetson Nano and Intel(R) Xeon(R) E5-2698 v4 CPU. For all comparisons, we show that FasterViT achieves a Pareto-front in for ImageNet Top-1 and throughput trade-off, hence validating the effectiveness and scalability of our model to different hardware platforms. In V100 ([Figure](https://drive.google.com/file/d/1FP0pDnZMdxw2Sy0z1tX-8A54T6eKD0cq/view?usp=sharing)) or TITAN RTX ([Figure](https://drive.google.com/file/d/1elnLkDMM5s2ch-E5MKb5YDj1vV_Rsx1s/view?usp=sharing)) comparisons for instance, we observe a strong performance for all FasterViT models. Even on CPU ([Figure](https://drive.google.com/file/d/1iVsev1XxgNaFEgFrhV0i4lGO-Fldl3z9/view?usp=sharing)), FasterViT still outperforms competitors by a significant margin. Please see the supplementary materials of the revised manuscript for all the figures regarding throughput and Top-1 accuracy. > **Additional Downstream Benchmarks** We thank the reviewer for their comment. We provide additional experiments for both object detection and semantic segmentation with more models, across different sizes, to demonstrate the effectiveness and efficiency of our work. Firstly, we present additional object detection experiments with DINO on MS-COCO dataset as shown below: | Backbone |Model|AP_box |Throughput |-------------------|----------------|---------------|---------------| |Swin-L |HTC++ |57.1|-| |Swin-L |DINO|58.5|71| |**FasterViT-4**|DINO|58.7|84| The DINO model with FasterViT-4 is 18.30% faster than its counterpart with Swin-L backbone in terms of image throughput and outperforms it by +0.1 in terms of box AP. We also added a semantic segmentation study on the ADE20K dataset with the FPN network, as shown below. Specifically, we compare against PoolFormer and PVT backbones: | Backbone |Model|mIoU |Throughput |-------------------|----------------|---------------|---------------| |PoolFomer-S36 |FPN |42.0|453| |**FasterViT-1**|FPN|42.7|491| |PoolFomer-M36 |FPN |42.4|368| |**FasterViT-2**|FPN|43.5|405| The model with FasterViT-1 backbone outperforms counterpart PoolFormer-S36 by +0.7 in terms of mIoU while also being 8.38% faster in terms of image throughput. Similarly, the model with FasterViT-2 backbone significantly outperforms PoolFomer-M36 counterpart by +1.1 in terms of mIOU while being 10.05% faster. We believe that the above experiments validate the effectiveness of FasterViT as an efficient backbone for downstream tasks such as segmentation and detection across different model sizes. > **How does latency change with respect to change of HAT parameters** Thank you for your question. We have conducted an additional ablation study to investigate the effect of HAT hyperparameters on the latency and accuracy of the model. Specifically, we examine the effect of window size (W) and carrier token window size (W_C) as shown below for 224x224 resolution. We use FasterViT-2 with 7x7 window size and 2x2 carrier token window size as the base model. Latency ratio is measured against this base model. | Resolution |Window Size|Carrier Token Window Size |Latency Ratio|Top1| |-------------------|----------------|---------------|---------------|---------------| |224 |7 |2|1|84.2| |224 |7 |1|1.05|83.9| |224 |7 |5|0.67|84.3| |224 |7 |6|0.57|84.4| |224 |7 |9|0.47|84.9| |224 |14 |0|0.9|84.4| We observe that increasing the carrier token window size can improve the performance at the cost of increased latency, sometimes by a significant margin. The 2x2 carrier token window size offers a great trade-off between accuracy and latency. In addition, increasing the window size from 7 to 14 increases the Top-1 accuracy by +0.2%. However, as expected, it increases the latency by 10%. We also note that although increasing the window size results in better performance, it does not scale properly to higher resolution images. As a result, HAT is a more effective and efficient mechanism that can be employed without sacrificing image throughput. > **How do you determine the architecture for the model variants from 0 to 6** Key design choices for each model variant are : (1) stage-wise depth (2) embedding dimension (3) number of attention heads. Our base model starts with a simple configuration that consists of 2, 3, 6, 5 layers per stage 1, 2, 3 and 4 respectively, an embedding dimension of 64 and 2, 4, 8, 16 attention heads for stages 1, 2, 3 and 4. From this base configuration, we increased the model capacity by increasing the embedding dimension or stage-wise depth to create a Pareto front of Top-1 and throughput trade-off.

Response to Reviewer B19F

We sincerely thank the reviewer for their time and efforts in reviewing our work and providing valuable feedback that can further strengthen our manuscript. Below please find our detailed responses: > **Intuition on why HAT and CT improve the model quality** HAT allows for modeling global contextual information with minimal impact on computational efficiency. Specifically, the carrier tokens in HAT facilitate capturing cross-region interactions in an efficient and scalable manner, especially for high-resolution images. Without carrier tokens, the self-attention can only process local information inside every window and lack the capability of capturing long-range information. > **Comparison of FasterViT with and without hierarchical attention for ImageNet** We have conducted an additional ablation study to demonstrate the effect of removing HAT for every FasterViT variant as shown below: | Model |HAT |Top-1 | |-------------------|----------------|----------------| | FasterViT-0 |Yes|82.1| | **FasterViT-0 wo HAT** |No|81.7| | FasterViT-1 |Yes|83.2| | **FasterViT-1 wo HAT** |No|82.8| | FasterViT-2 |Yes|84.2| | **FasterViT-2 wo HAT** |No|83.7| | FasterViT-3 |Yes|84.9| | **FasterViT-3 wo HAT** |No|84.2| | FasterViT-4 |Yes|85.4| | **FasterViT-4 wo HAT** |No|84.3| We observe that removing HAT can significantly decrease the accuracy, especially for larger variants such as the FasterViT-4 model. In addition to the above experiments, Figure S.1 offers more insights on the effectiveness of HAT. Specifically, in this figure, the carrier tokens have been appended to the left top corner. The vertical line in dense attention maps indicates that all other tokens append to the carrier tokens, hence allowing for cross-region communications to capture both short and long-range spatial dependencies. > **Significance of HAT and plug-and-play module gains in Swin Transformers** Swin Transformers use window shifting to capture the local cross-region interactions. However, this scheme is sub-optimal as it is limited to a small area of coverage which cannot effectively capture the long-range spatial dependencies. However, HAT proposes an efficient and effective way of modeling global information which is important for downstream tasks and high-resolution images. The improvements shown in Table 7 are significant as no additional hyper-parameter tuning was done. Specifically, as shown in In Table 7, using HAT with Swin improves the performance by +0.9 in terms of mIoU for semantic segmentation. Similarly, it improves the performance by +0.5 and +0.6 interim of box AP and mask AP for object detection and instance segmentation. In addition, FasterViT-4 outperforms counterpart Swin-L model by +0.2% when fine tuning on 384x384 resolution. > **Throughput numbers and HAT overhead in Table 7** We provide throughput numbers for experiments in Table 7 in the following: | Model |Top-1 |Throughput (ImageNet) |AP_box|AP_mask|Throughput (det)|mIoU|Throughput (seg)| |-------------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------| | Swin-T |81.3|2758|50.4|43.7|161|44.5|330| | **Swin-T+HAT** |81.7|2721|50.9|44.3|150|45.4|338| As seen above, HAT has a minor impact on overhead across different tasks and can be used as a viable standalone self-attention mechanism. We have also added throughput numbers in Table 7 of the revised manuscript (all numbers are shown in red). > **Throughput comparison on other platforms** We have added additional throughput comparison on different platforms such as V100, TITAN RTX and A6000 GPUs, Jetson Nano and Intel(R) Xeon(R) E5-2698 v4 CPU. For all comparisons, we show that FasterViT achieves a Pareto-front in for ImageNet Top-1 and throughput trade-off, hence validating the effectiveness and scalability of our model to different hardware platforms. In V100 ([Figure](https://drive.google.com/file/d/1FP0pDnZMdxw2Sy0z1tX-8A54T6eKD0cq/view?usp=sharing)) or TITAN RTX ([Figure](https://drive.google.com/file/d/1elnLkDMM5s2ch-E5MKb5YDj1vV_Rsx1s/view?usp=sharing)) comparisons for instance, we observe a strong performance for all FasterViT models. Even on CPU ([Figure](https://drive.google.com/file/d/1iVsev1XxgNaFEgFrhV0i4lGO-Fldl3z9/view?usp=sharing)), FasterViT still outperforms competitors by a significant margin. Please see the supplementary materials of the revised manuscript for all the figures regarding throughput and Top-1 accuracy. > **Small text in figures** Thank you for your suggestion. We have improved the figures and enlarged their embedded text.