Hybrid Mamba for Few-Shot Segmentation

Summary

This work introduces Mamba to FSS. Particularly, it indicates two issues suffered by the original Mamba when being applied to the cross attention case, namely, the support forgetting issue and the intra-class gap issue. To address them, a hybrid Mamba block (HMB) is designed, which can effectively incorporate query FG features with the support FG features, thereby leading to better FSS performance

Strengths

1. HMB introduces Mamba into FSS, which reduces the computational cost from quadratic complexity to linear complexity. 2. The illustration of HMB is clear and easy to understand. 3. Experiments show the method can achieve state-of-the-art.

Weaknesses

1. Introducing mamba into FFS is to reduce the computational cost, but no experimental results support it. Comparing the efficiency of the proposal method with it of cross-attention based methods is needed. 2. Writing is not standard somewhere. For instance, few-shot segmentation includes few-shot semantic segmentation and few-shot instance segmentation, but the work didn't refer to any instance segmentation task. 3. The content in coco 'train_data_list' file of the supplementary code leaks your identity. I don't know if it violates the double-blind rule.

Questions

My questions are written in the above Weakness.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

This work addresses the limitations(computational cost) in theory， but no experimental results can prove it. Besides, this work don't have negative societal impact.

Summary

This paper proposes a hybrid mamba network (HMNet) to capture inter-sequence dependencies for few-shot segmentation tasks (FSS). The authors identify two issues when applying the original Mamba network to cross-attention scenarios: the support forgetting issue and the intra-class gap issue. The proposed HMNet includes: 1) Support Recapped Mamba (SRM), which periodically recaps the support features when scanning the query to ensure the hidden state contains rich support information; and 2) Query Intercepted Mamba (QIM), which intercepts the mutual interactions among query pixels when propagating the hidden state (with pure support FG) to encourage the fusion of more information from support features. Extensive experiments on PASCAL-5^i and COCO-20^i demonstrate the superiority of the proposed method.

Strengths

1. This paper indicates the support forgetting and intra-class gap issue when applying mamba to FSS. 2. The proposed HMB is more effective for the fusion of support-query features, leading to improved FSS performance. 3. Experimental results on PASCAL-5^i and COCO-20^i verify its effectiveness.

Weaknesses

1. The Mamba model exhibits significantly enhanced efficiency over the attention-based models as the sequence length increases, such as the million-length real-world data mentioned in the Mamba paper. In FSS, is there a comparative analysis that evaluates efficiency and effectiveness across different patch quantity settings? 2. The proposed SRM strategy includes inserting support patches periodically among query patches. Could this approach inadvertently lead to forgetting the query patches in earlier positions? 3. For consistency in terminology, consider distinguishing between "feature" and "patch," as in the caption of Figure 3: "...alternatively appeared support features and query patches...".

Questions

See above.

Rating

5

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

It would be beneficial if the author could address the potential risk of query patch forgetting and include comparisons with attention-based methods across different patch settings.

Summary

This paper proposes HMNet for FSS, which addresses the issues of support forgetting and intra-class gap in few-shot segmentation by designing support recapped Mamba and query intercepted Mamba, thereby utilizing support information more effectively to enhance segmentation performance. The authors evaluate HMNet through extensive experiments on two public benchmark datasets.

Strengths

1. This paper is the first attempt to apply Mamba to the FSS task. 2. Two mechanisms are designed to adapt the original Mamba to better enhance the query FG features. 3. The proposed method achieves SOTA performance.

Weaknesses

1. The core idea of HMNet appears to be rearranging and combining the support and query features as inputs to Mamba, which may lack a prominent contribution to the design of novel Mamba. 2. To avoid the support forgetting issue, why not use two vanilla Mamba (similar to self-attention) blocks to extract support and query features separately, and then use another Mamba to interact with them (like cross-attention)? Such a mechanism has been evaluated as effective (like HDMNET, CVPR 2023).

Questions

In few-shot segmentation, the background information of support samples may also be helpful in identifying the background regions in the query samples. Have the authors tried to incorporate this aspect into HMNet?

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations of the paper.

Summary

This work proposes a hybrid Mamba network for few-shot segmentation, including a support recapped Mamba to periodically recap the support features when scanning query and a query intercepted Mamba to forbid the mutual interactions among query pixels.

Strengths

The idea of adapting Mamba for few-shot segmentation is good. The performance looks good.

Weaknesses

1. The motivation of using mamba for FSS is unconvincing. The authors state that they use mamba for FSS is because that mamba's complexity is only linear. However, the proposed mamba-based method is used only for fuse support foreground (FG) into query features. There are also many light-weight feature fusing methods for FSS and show good performance and efficiency. The authors need to provide convincing reasons how the mamba works better than other feature fusing methods. Otherwise, just applying a new technology to FSS without deep insight is not acceptable. 2. The experimental analysis is insufficient. First, the comparison experiments between the proposed mamba methods and other feature fusing methods are missing. I cannot know if the proposed method is better, on performance or efficiency. I mean the fair comparison on a same baseline method to compare with other methods, rather than just the Table 1 and Table 2. The overall performance comparison is unfair because the baseline methods are different. Second, the analysis is insufficient. The authors state that the proposed mamba methods solves the support forgetting and intra-class gap issue problems. But how these problems are solved? The Table 3 and Figure 5 just show that the proposed method achieves better performance. However, it is still unclear how the proposed method solve these problems. 3. It also lacks the comparison with SAM-based FSS methods.

Questions

See weakness

Rating

4

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

No.

Request for Discussions

Dear Reviewers, ACs, SACs and PCs, Thanks again for your detailed review and constructive suggestions for improvement. We have carefully considered your comments and suggestions, and provided detailed responses. As the discussion period is concluding in 3 days, we kindly request you to read our rebuttal at your earliest convenience. We are hopeful that our responses adequately address your concerns, and you can reconsider your score. If you require further clarification on any points, please do not hesitate to reach out. We are willing to provide additional information to ensure your confidence in the responses made. Sincerely, Authors