Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model

Summary

This paper presents a multi-scale Vmamba model, which incorporates multi-scale information into the design of the Vmamba architecture. Additionally, the authors analyze how the attenuation coefficient between tokens increases as the modeled distance in Vmamba grows, whereas MSVmamba alleviates this attenuation issue by reducing the sequence length. The authors have conducted extensive experiments to thoroughly validate the effectiveness of MSVmamba. Moreover, the authors have also employed SE modules and ConvFFN to further enhance the model's performance.

Strengths

1. The author's writing is clear and easy to follow. 2. The authors analyzed the issue of long-distance forgetting and proposed an effective method to address this problem. 3. The authors conducted comprehensive ablation experiments to validate the role of each module (MS2D, SE, ConvFFN) in MSVmamba for classification tasks.

Weaknesses

1. Although MSVmamba has achieved better results compared to other Vmamba models, its scalability has not yet been validated. Could the authors provide training results of MSVmamba on larger models, such as those with 50M and 90M parameters, to demonstrate the model's scalability? 2. The authors' ablation experiments only validated the performance of the modules in classification tasks. It would be better to further verify the roles of each module in more fine-grained tasks such as detection and segmentation.

Questions

Please refer to the weakness

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper introduces a Multi-Scale Vision Mamba (MSVMamba) for computer vision tasks. It uses a multi-scale 2D scan operation on both the original and sub-sampled features to preserve long-range information and reduce computational costs. In addition, they address the problem of channel mixing in Mamba-based models by introducing a Convolutional Feed-Forward Network (ConvFFN) module. The resultant model achieves favorable performance on image classification and a variety of downstream tasks such as detection and segmentation.

Strengths

1. The paper is well-written and easy to follow. 2. It attempts to address an important issue in making Mamba-based vision models more suitable for computer tasks. In order to achieve this, it introduces a hierarchical architecture, although not novel, but also integrates various modules such Convolutional Feed-Forward Network (ConvFFN) to further enhance the performance. 3. The analysis of effectiveness in 2D selective scan approach, initially introduced in VMamba, seems to be interesting and provide further insights on how to improve scan for vision tasks.

Weaknesses

1. The major issue with this work is lack of novelty. Hierarchical Mamba-based vision models have been already introduced in VMamba and also its other variants. The MSVSS block seems to be a minor improvement over the existing VMamba block. In addition, the role of ConvFFN seems to be quite marginal. This is due to the fact that MLP blocks themselves can inherently perform channel mixing to a great extent. 2. Experiments are insufficient. This work only presents three small variants of MSVMamba-N, MSVMamba-M and MSVMamba-T with the biggest model having only 33M parameters. Hence, it is not really clear how the proposed approach scales for mid to larger sized models which have better accuracies. It even can't be compared to the small variants of many models (e.g. Swin-S) due to its contrived setting. 3. The paper only focuses on number of FLOPs as a representative for efficiency. However, a more practical scenario involved measuring throughput (or latency) on different devices (GPU, TPU, etc.). In particular, it is important to understand if the 2D selective scan approach introduces any significant overhead. Post-rebuttal: The following issues and weaknesses were revealed during rebuttal after interactions with the authors: 1. The proposed MSVMamba is slower than models such as ConvNeXt in both smaller and higher resolutions in terms of throughput (see Table 1 and Table 2 in rebuttal). This limits the practical usage of this work due to its lower throughput and can present significant challenges. 2. The authors deliberately presented results from the first version of VMamba which is not optimal. Although the second version (https://arxiv.org/abs/2401.10166v2) was released 42 days before the submission deadline, the authors claim that it should be considered as concurrent. This argument is not well-founded since we need to fairly evaluate the contribution of this work against VMamba and other methods. 3. The issue of limited novelty presents itself when comparing against VMamba (both first or second iterations). As expected, MSVMamba does not significantly improve the results -- authors claimed that MSVMamba addressed the long-distance forgetting issue in VMamba which is not backed up by these results. Considering these issues, I lower my score to strong reject (2). I encourage the authors to revise their manuscript, include best results from VMamba and try to evaluate the contributions of their work quantitatively.

Questions

1. How does the model compare in terms of image throughput to other Mamba-based vision models as well as CNN-based and ViT variants on a GPU ? 2. Is the 2D selective scan approach faster than the naive selective scan that is originally introduced in Mamba ? 3. Did authors try to scale up the model size to observe if performance is comparable to other models ?

Rating

2

Confidence

5

Soundness

2

Presentation

2

Contribution

1

Limitations

Yes

Summary

This paper presents a multi-scale vision mamba aimed at improving the performance of state space models (SSMs) in vision tasks while maintaining efficiency. The motivation stems from analyzing the multi-scan strategy in vision mamba, where the authors link its success to alleviating the long-range forgetting issue of SSMs in vision tasks. To address this problem effectively, the authors propose a multi-scale 2D scanning technique on both original and downsampled feature maps, which reduces the number of tokens in the multi-scan strategy. This method enhances long-range dependency learning and cuts down computational costs. Additionally, a ConvFFN is incorporated to overcome channel mixing limitations. Experimental results across various benchmarks validate the proposed multi-scale vision mamba's effectiveness.

Strengths

The analysis of the multi-scan strategy's success in vision mamba is intriguing and could be valuable to the research community. The proposed approach is straightforward yet effective, addressing the long-range forgetting problem while significantly reducing computational costs. The approach strikes a better balance between performance and FLOPs, as demonstrated in the experiments. The paper includes comprehensive experiments on widely-used datasets and tasks, with comparisons to leading neural architectures showing the proposed networks' superiority.

Weaknesses

The authors should consider including some simple baselines in additional ablation studies. For instance, besides the half-resolution branches in the proposed MSVMamba, reducing the scanning number in vision mamba would be a useful baseline. Incorporating these simple baselines could further highlight MSVMamba's effectiveness. The paper outlines that the proposed MSVMamba uses 3 half-resolution branches and 1 full-resolution branch (Equations 8-11). However, the scanning direction for these branches is not clearly described. There is a lack of discussion on how the scanning direction for the full-resolution branch is chosen and how this choice impacts performance.

Questions

Could you provide more comparisons with simple baselines? Could you clarify the selection of different branches concerning the scanning direction and discuss its impact?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Reflecting on the weaknesses mentioned.

Summary

The paper introduces a novel vision backbone model, MSVMamba, which incorporates State Space Models (SSMs) to address limitations in computational efficiency and long-range dependency capture in vision tasks. The model utilizes a multi-scale 2D scanning technique and a Convolutional Feed-Forward Network (ConvFFN) to improve performance with limited parameters.

Strengths

The paper is easy to follow.

Weaknesses

1. Lack of novelty. The paper propose MSVMamba, the main contribution is shown in table.4. MS2D, SE, ConvFFN have already been proposed in previous papers. The main improvements come from existing knowledge. 2. Speed The paper does not systematically measure the speed of their model on all the tasks and scales. The training and inference of MSVMamba could be slow compared with current models, like CAFormer, Conv2Former, CSwinTransformer. 3. Scalability The model in the experiments is relatively small. Model with more than 300M or 600M parameters could show whether MSVMamba does perform better than current sota model. Now the experiments just show that the model converges quickly w.r.t. model size.

Questions

Please refer to the weakness.

Rating

2

Confidence

4

Soundness

2

Presentation

2

Contribution

1

Limitations

Please refer to the weakness.

Summary

The paper introduces Multi-Scale VMamba (MSVMamba), a novel vision backbone model that leverages State Space Models (SSMs) to address the challenges of quadratic complexity in Vision Transformers (ViTs). The proposed Multi-Scale 2D Scanning (MS2D) and Convolutional Feed-Forward Network (ConvFFN) contributes to the final performance.

Strengths

- This paper presents a multi-scale design of VMamba, reducing the sequence length by downsampling the input features. - The SSM module proposed in this paper incorporated the strength of residual, dwconv, ssm, se-block. - This paper achieves good results on classification and dense prediction benchmarks.

Weaknesses

- This paper lacks the efficiency comparison among training/inference FPS and memory usage on real GPU devices. - The proposed multi-resolution processing seems to contain too many inductive biases. I wonder if the gains would be reduced when applying to a standard-ablation-size model, i.e., VMamba-T.

Questions

Please refer to the weakness

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes

Discrepancies in VMamba Benchmarks

I'd like to thank the authors for providing the rebuttal. However, upon reading Table 1, VMamba Top-1 accuracy benchmarks seem to be lower than the official release (and their paper): https://github.com/MzeroMiko/VMamba Any reason for this discrepancy ?

Comment by Reviewer

I appreciate the authors' response, but why not using the best results/models when comparing to them ? The first version in Jan 18 was a preliminary effort which may not be optimized. And our goal is not to just show better performance, but also provide a meaningful comparison to previous efforts. The results I refer to can also be found in the second iteration of their arXiv paper which has been released since 10 Apr 2024: https://arxiv.org/pdf/2401.10166v2

Reviewer's Response to Rebuttal

I would like to thank the authors for providing responses to my feedback as well as uploading the rebuttal. I have the following concerns: 1. As mentioned in my other comment, the reported Top-1 accuracy from VMamba seem to be lower than their existing benchmarks. The authors claim that they have used the results from the first iteration of the VMamba arXiv submission. However, the benchmarks I refer to can also be found in the 2nd iteration which was available since Apr 10: https://arxiv.org/pdf/2401.10166v2 Can the authors provide an updated version of Table 1 for VMamba and MSVMamba models alone (Tiny, Small and Base) with the updated Top-1 accuracy as reported in the above arXiv submission ? it should suffice to just reply to my comment with this table. 2. The reported numbers in Table 2 for efficiency comparison are for batch size 32. Why not report numbers for standard 224x224 resolution with batch size 128 which matches the same setup in Table 1 ? Our goal here is to have a fair comparison to previous models to further understand the contributions of the proposed effort.

Further Comment by Reviewer

I thank the authors for providing responses to my feedback. Regarding the throughput comparison, the authors mention: " With the input resolution increase, the memory cost for models also increase significantly". However, my request was to use an input resolution of **224x224** and batch size of **128** which is very standard and has been presented in previous works before. This setup should be achievable on any consumer GPUs with moderate VRAMs. Can the authors only provide a comparison between MSVMamba, ConvNeXt and Swin Transformers in terms of throughput using this setup ? I would like to understand how your model compares to well-established models in the literature.

final reviewer

Thanks the author's efforts. My concerns are well addressed. I will keep my initial rates (7).

Thanks for addressing my concerns

I'd like to thank the authors' response to my question. I think the response is clear enough to address my confusion.

Please allow me to step forward and share my thoughts on this discussion thread

VMamba is a new and promising backbone network in computer vision that warrants more research efforts. However, from what I understand, VMamba-based models currently face challenges in achieving better efficiency than CNNs and ViTs. This is why numerous recent works aim to further enhance Mamba to make it viable for real-world applications. In my opinion, the authors’ claim of achieving a 1.5x speedup in inference and a 2.0x speedup in training is commendable and meets expectations. I reviewed the NeurIPS 2024 FAQ for Authors on the official website, and it appears that the second version of VMamba, which appeared on arXiv this April, can indeed be considered concurrent with this submission. Therefore, I have no concerns about the current comparative experiments, and it’s great to see the authors include new comparison results with that second version of VMamba. Overall, I am inclined to accept the paper and am open to further discussions with other reviewers.