**We sincerely appreciate the feedback from the reviewer. We address the newly raised questions below.**
---
**Q1. The actual speed-up of the proposed method seems less ideal than that of the ViT pruning method like DynamicViT. I am still concerned about the value of studying token pruning for SSM-based models.**
Token pruning is crucial for enhancing efficiency, especially with long sequences (please refer to our answer to **Q1** of Reviewer **M2Lk**). By reducing the number of tokens processed, we can significantly decrease computational cost and memory usage, thereby improving the scalability of SSMs for real-world applications. In this work, we introduce a general token pruning method specifically designed for SSM-based vision models. This is the **first successfully** token pruning work to **effectively** handle this problem on SSMs with **huge performance improvement**.
Given the limited time (less than one day), we conducted an acceleration comparison between our method and DynamicViT under the same pruning ratio on SSMs. The results are as follows:
|Model| Method | FLOPs(G)| Throughput |
|-|-|-|-|
| ViM-S |Ours| 3.60 | 1.27×|
| ViM-S |DynamicViT| 3.56 | 1.30×|
| PlainMamba-L3 |Ours|8.44| 1.43×|
| PlainMamba-L3 |DynamicViT| 8.37 | 1.46×|
As shown, there is minimal difference in acceleration under the same pruning ratio when compared to methods like DynamicViT on SSMs. However, our method provides **much higher** accuracy compared to ViT methods. For example, recent works such as ToMe [1] and LTMP [2], which we discussed in our response to **Reviewer TrFe**, highlight that our method achieves **significant performance improvements** over token pruning methods in ViT. The results are as follows:
|Methods | FLOPs(G) | Top-1 Acc. (%)|
|------------|-----|-----|
| ViM-T |1.50| 76.1 |
| ViM-T-ToMe | 1.28|71.6|
| ViM-T-LTMP | 1.29|72.2|
| ViM-T-EViT | 1.28|71.3|
| ViM-T-Ours | 1.29 |75.1|
---
**Q2. EViT is a token pruning baseline published in 2022. Recent methods like AViT [r1] and STViT [r2] can achieve no accuracy drop after pruning. The results presented in the paper are relatively weak.**
While different token pruning methods in ViT may prune tokens without reducing accuracy, they all face the **same challenges** when applied to SSMs. In SSM, the relationships between tokens are **different** from those in ViT. SSMs handle sequential data, where the dynamic characteristics and contextual dependencies of each token determine its importance. Simple pruning might disrupt the sequence's integrity or the transmission of crucial information, leading to **significant** performance **degradation**. We would like to highlight that our work is the first successful application of token pruning to SSMs, resulting in substantial performance improvements.
Due to the limited time (less than one day for applying AViT and STViT), we summarize the more recent work like ToMe [1] and LTMP [2] from our answer to **reviewer TrFe**. We will add these results into our revised version. We implemented ToMe[1] as well as LTMP [2] for our SSM-based models to provide a comprehensive comparison with state-of-the-art techniques. As demonstrated in the following table, our method can outperform all baselines with non-marginal improvements.
|Methods | FLOPs(G) | Top-1 Acc. (%)|
|------------|-----|-----|
| ViM-T |1.50| 76.1 |
| ViM-T-ToMe | 1.28|71.6|
| ViM-T-LTMP | 1.29|72.2|
| ViM-T-EViT | 1.28|71.3|
| ViM-T-Ours | 1.29 |75.1|
**Overall, we would like to emphases that regardless of the ViT token pruning method, if it does not address the specific characteristics of SSMs, it will not achieve the same level of effectiveness in SSMs. Therefore, it is unreasonable to compare the accuracy drop level on ViT with it on SSMs.**
[1] Token Merging: Your ViT But Faster, ICLR 2023
[2] Learned Thresholds Token Merging and Pruning for Vision Transformers, TMLR 2023