iFormer: Integrating ConvNet and Transformer for Mobile Application

Response to Reviewer aPLf

We thank the reviewer for the positive feedback! > Q1: What is the runtime complexity of iFormer network? Given an input $\ \mathbf{x}\in \mathbb{R}^{C\times H\times W}$ and a window size of P $\times$ P, as detailed in Section E, the computational complexity of iFormer is as follows: $\Omega (\text{SHMA}) = 4HWC^2 \text{(QKV and output projection)} + HWC \text{(element-wise product of modulation)} + 2P^2HWC \text{(self-attention)}$, $\Omega (\text{FFN}) = 8HWC^2.$ In image classification, we do not utilize window attention since the feature size is 14$\times$ 14 in stage 3 (it equals to the window attention when P=14). In downstream tasks, we adopt a window size of P=16. > Q2: When running on iPhone (mobile device), what is the peak memory consumption? Currently, we employ the Xcode benchmark tool to evaluate the latency of our models. However, this tool does not support detailed memory consumption. We are actively exploring alternative methods to monitor the peak memory usage of iFormer on the iPhone. > Q3: How long the iPhone charge will last if an iFormer based app is run on certain fps? We appreciate your suggestions, and we plan to develop an app that provides more comprehensive information when running iFormer on the iPhone, including estimated battery life, peak memory usage, and other relevant metrics.

Response to Reviewer fhTG

We would like to express our gratitude to the reviewer for their insightful comments and questions. > Q1: How does this method compare with neural architecture search (NAS) methods? In our study, we have compared two NAS methods, namely MNV4-Conv and EfficientFormerV2. iFormer demonstrated superior performance as depicted in Tables 3 and 4. NAS relies on the pre-defined search space of operators with the goal of identifying the optimal combinations and configurations of various operators. The innovative and efficient SHMA block significantly outperforms existing operators, enabling iFormer to achieve a better trade-off between accuracy and latency. As a general methodology, NAS can also be applied to iFormer and propel it towards a new SOTA. > Q2: How does the designed model perform on other mobile devices, such as NVIDIA Jetson Nano or Raspberry Pi? Due to the limitation of available devices and time constraints, we have not evaluated iFormer on a broader range of devices. In the future, we plan to expand the evaluation of iFormer to more platforms, including NVIDIA Jetson Nano and Android devices. It is important to note that designing a model performing well across multiple hardware platforms is challenging, as varying memory and computational architectures can lead to different behaviors.

Response to Reviewer waKF (Part 1/2)

W1: We would like to clarify that iFormer is not a simple concatenation of multiple existing methods. Instead, we propose several novel designs aiming at lightening ConvNeXt. > Single head self-attention has been conducted in "Shvit: Single-head vision transformer with memory efficient macro design" . 1. Single Head Attention (SHA) differs from its counterpart in SHViT in the following aspects: **In terms of motivation**, iFormer explores efficient attention mechanisms specifically tailored for the on-device environment, whereas SHViT is geared towards general-purpose GPUs, which may exhibit different hardware characteristics. **In terms of methodology**, we utilize single head attention with more channels, while SHViT employs fewer than 1/4 of channels for attention. The reduced number of channels can result in a lower rank of the attention matrix, potentially degrading its expressiveness. Additionally, the split and concatenate operations in SHViT introduce extra runtime. The experimental results in Table 15 show that iFormer achieves a better trade-off than SHViT. We also conduct a more fair comparison with SHViT. We use the SHA baseline in Table 1, specifically denoted as 'SHA' in Figure 2. Then we move toward SHViT as follows: | Model | Params | GMACs | Latency | Top-1 | |:----------------------------|:------:|------:|--------:|------:| | SHA Baseline without Modulation | 9.9M | 1758M | 1.12ms | 79.8 | | + split | 9.9M | 1758M | 1.18ms | - | | + attention on 1/4 channels | 8.3M | 1547M | 1.02ms | - | | + concat | 8.7M | 1579M | 1.11ms | 79.5 | The discussion can also be found in the Section D of the revised version (highlighted in blue). It can be observed that split and concat operations introduce additional runtime. Moreover, the SHA of iFormer demonstrates stronger performance than the counterpart of SHViT under similar latency (79.8 v.s. 79.5). This improved performance may be attributed to the greater number of channels in the attention mechanism > Alternative to standard self-attention has been conducted in GhostNetV2. 2. SHMA is fundamentally different from the attention mechanism in GhostNetV2, which decomposes spatial interactions into horizontal and vertical interactions and behaves more like the Mlp-mixer [1]. In contrast, SHMA integrates the single head self-attention with modulation. Besides, the experiments in Table 3 suggest that iFormer demonstrates a superior and more efficient alternative to self-attention than GhostNetV2 (with V3 being the upgraded version). > Modulation in the token mixer module has been conducted in Conv2Former. 3. Thanks for bringing Conv2Former to our attention, we include discussions about it in the related work in the revision. Modulation is a general methodology for enhancing dynamic interactions. SHMA distinguishes itself as a novel approach from existing modulation techniques including Conv2Former, VAN, and FocalNet in the direct integration of self-attention into the context branch of modulation. This integration enables SHMA to capture more informative context. [1] Mlp-mixer: An all-mlp architecture for vision. NeurIPS 2021.

Response to Reviewer waKF (Part 2/2)

W2 and Q: Although iFormer is designed for mobile-device applications, the combination of fast local representation capacity of convolution and the efficient global modeling proficiency of the proposed SHMA enables its scalability for a broader range of applications. To demonstrate the scalability of iFormer, we developed a larger model named iFormer-H with 99M parameters and trained it for 300 epochs following the same strategy outlined in Section B of the revision (highlighted in blue). It is important to note that we add drop path and layer scale, which are commonly used in the training of larger models [1,2,3]. The performance results are provided as follows: | Model | Params | GMACs | Top-1 | |:------------------|:------:|------:|------:| | ConvNeXt-Base[1] | 89M | 15.4G | 83.8 | | TransNeXt-Base[2] | 90M | 18.4G | 84.8 | | iFormer-H | 99M | 15.5G | 84.8 | | MaxViT-Base [3] | 120M | 24.0G | 84.9 | A highlight from the results is that iFormer is not specifically designed or trained for this scale. Despite this, iFormer-H outperforms ConvNeXt, achieving a 1.0% increase in accuracy while maintaining a similar number of FLOPs. Additionally, it demonstrates comparable performance to TransNeXt-Base, despite utilizing fewer FLOPs. These findings indicate the potential for broader applications of iFormer. We will also open source this model with source code in the Github and plan to explore larger models suitable for mobile devices in future work. [1] A ConvNet for the 2020s. CVPR 2022 [2] TransNeXt: Robust Foveal Visual Perception for Vision Transformers. CVPR 2024 [3] MaxViT: Multi-Axis Vision Transformer. ECCV 2022

Response to Reviewer XLkf

We appreciate the reviewer's insightful comments and questions! Q1: We apologize for the mistake. The latency of RepViT-M1.0 is 1.54 ms, rather than 1.64 ms. We have updated this information in Tables 3 and 4 in the revision (highlighted in blue). Additionally, we have conducted a thorough review of all experimental results included in the paper. Q2: We have included these two papers in the related work section and included a comparison of CMT in Table 15. These updates are highlighted in blue in the Section 2.2 Efficient Vision Transformers of the revision.

Response to Reviewer wKqn (part 1/2)

We thank the reviewer for the thorough and constructive feedback. W1 and Q1: We would like to clarify that SHA serves as an intermediate design towards SHMA. So changing the convolutional blocks yields a 1.0% improvement. > I'd like to know further explanation for this design and ablation studies ... if possible. We conduct ablation studies on the choice of Conv versus ViT blocks, which ultimately lead to the architecture of iFormer. We choose the model after enlarging the kernel size as a start point, then we progressively replace the convolutional blocks in Stages 3 and 4. We do not modify Stages 1 and 2 as they have larger spatial dimensions, which significantly increase memory requirements for the self-attention mechanism. We present these findings here and have incorporated them into the Section 5 (Choice of Conv v.s. ViT Blcoks) of the revised manuscript, highlighted in blue. | Model | Params | GMACs | Latency | Top-1 | |:-----------------------------------------------------------------|:------:|------:|--------:|------:| | Baseline | 9.4M | 1760M | 1.00ms | 79.4 | | Replacing 22% Conv Blocks in Stage 3 as SHA | 9.1M | 1724M | 1.02ms | 79.5 | | Replacing 22% Conv Blocks in Stage 3 as SHMA | 9.2M | 1739M | 1.04ms | 79.6 | | Replacing 50% Conv Blocks in Stage 3 as SHA | 8.8M | 1689M | 1.04ms | 79.5 | | Replacing 50% Conv Blocks in Stage 3 as SHMA | 8.9M | 1712M | 1.07ms | 79.8 | | Replacing 78% Conv Blocks in Stage 3 as SHA | 8.3M | 1635M | 1.12ms | 79.3 | | Replacing 78% Conv Blocks in Stage 3 as SHMA | 8.5M | 1685M | 1.17ms | 79.6 | | Replacing 100% Conv Blocks in Stage 3 as SHA | 7.9M | 1599M | 1.17ms | 78.1 | | Replacing 100% Conv Blocks in Stage 3 as SHMA | 8.3M | 1665M | 1.25ms | 79.0 | | Replacing 100% Conv Blocks in Stage 3 as SHMA and 100% in Stage 4 | 10.0M | 1792M | 1.15ms | 80.4 | Since Stage 4 contains only two blocks, we do not conduct further split for the ratio. As shown in the above table, the ViT block incurs more runtime. By replacing half of the convolutional blocks in the third stage and all blocks in the final stage, we achieve a pretty trade-off between accuracy and latency.

Response to Reviewer wKqn (part 2/2)

W2, W3: We are sorry for any confusion caused by the unclear statement. We would like to clarify that our SHA has no relationship to SHViT. We have included more details about SHA in the Section D of the revised version (highlighted in blue). Additionally, we will release the source code in the final version. > According to the citation of SHViT in this paper, I suppose ... in SHViT design. The SHA in iFormer is different from SHViT in the following three key aspects. First, in terms of motivation, iFormer explores efficient attention mechanisms in an on-device environment, while SHViT focuses on general-purpose GPUs, which may exhibit distinct characteristics. > full channels of input (CxHxW) are ... single head. Second, in terms of methodology, the full channels of the input (CxHxW) are projected to Q/K (C/R X L) and V (C x L), where R is the head reduction ratio and is set to 2 in our models. We have updated the Figure 4 of the revision to reflect this. In contrast, SHViT splits the input and utilizes fewer than 1/4 of the channels for attention. This reduction in channels can lower the rank of the attention matrix, thereby degrading its representational capacity. Furthermore, we do not use split and concatenation operations as they tend to exacerbate memory access costs. > I suggest the authors conduct an ablation study ... the proposed method. To show this, we refer to the SHA baseline in Table 1, i,e., 'SHA' in Figure 2. Subsequently, we transit it toward SHViT as follows: | Model | Params | GMACs | Latency | Top-1 | |:----------------------------|:------:|------:|--------:|------:| | SHA Baseline without Modulation | 9.9M | 1758M | 1.12ms | 79.8 | | + split | 9.9M | 1758M | 1.18ms | - | | + attention on 1/4 channels | 8.3M | 1547M | 1.02ms | - | | + concat | 8.7M | 1579M | 1.11ms | 79.5 | It can be observed that split and concat operations introduce additional runtime. Furthermore, the performance of the SHA in the SHViT exhibits a decline compared to its counterpart in iFormer under similar latency conditions (79.8 v.s. 79.5). This degraded performance may be attributed to the reduced number of channels in the attention mechanism. We have updated this information in the Section D of revision. Q2: Since we do not apply SHViT, the only difference between the MHA Baseline and the SHA Baseline in Table 1 is the number of heads, which pertains to reshaping. The performance and latency of the MHA serve as the missing step between 'kernel sz.' and 'SHA', represented by the MHA Baseline.