Focus on Query: Adversarial Mining Transformer for Few-Shot Segmentation

Summary

The paper argues that mining information from the support set does not effectively improve Few-shot Segmentation (FSS) results. Instead, the paper proposes an query-centric FSS model called AMFormer, which uses adversarial learning to generate segmentation results with only rough support guidance.

Strengths

1) The motivation in this paper is both convincing and novel. 2) The paper is well-organized and easy to follow. 3) The proposed AMFormer achieves SOTA performance on the two datasets.

Weaknesses

Major Weaknesses 1) Overpowered baseline model In Tab5, the results of baseline model have surpassed most previous works, and the main module of this paper improves on the baseline by 5.7%. The authors should provide more details on the baseline model and explain why it achieves such good performance. (Earlier works use a similar ‘’MAP + cosine similarity’’ design [1*, 2*] and only achieve 45%~55% mIoU on Pascal) In addition, in L282, the authors seem to add the cycle-consistant-transformer to the baseline model, which is not introduced in Sec. 3 and Fig. 3. I suggest that the authors use only the Discriminative Region Localization model (introduced in Sec. 3.3.2) as their baseline model. Minor Weaknesses 1) Running time & inference speed The proposed AMFormer exploits multi-scale transformers in G and D, which may increase the computational costs. Please discuss the impacts of different components brought to the model efficiency. 2) Learnable parameters in Tab. 4 Please check the learnable parameters in Tab. 4, e.g. HSNet, which is inconsistent with the corresponding paper. 3) More recent works More recent works [3*, 4*, 5*] should include in Sec. 2.2 and Tab2, 3. 4) Generalization study of G and D G and D should be easily applied to other methods, e.g., using an advanced method to generate the High-Confidence Map (Mqt), rather than Discriminative Region Localization model. Therefore, it is suggested the authors to investigate more about the G and D in different methods (HDMNet, BAM) to prove its effectiveness is generalizable enough. [1*] PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment, ICCV 2019 [2*] SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation, Transactions on Cybernetics 2020 [3*] Mask Matching Transformer for Few-Shot Segmentation, NeurIPS 2022. [4*] Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning, NeurIPS 2022. [5*] Feature-Proxy Transformer for Few-Shot Segmentation, NeurIPS 2022.

Questions

Although AMFormer achieves SOTA performance, I am confused about the results, so I give a rating of 4. I will increase the rating if the authors address all my concerns and questions.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

More training epochs comparing with some previous methods.

Summary

This work proposes a new query-centric FSS model Adversarial Mining Transformer for few-shot segmentation. However, the idea is exactly same as SSP (ECCV2022).

Strengths

This work proposes a new query-centric FSS model Adversarial Mining Transformer for few-shot segmentation.

Weaknesses

The idea is exactly the same as the self-support idea of SSP (ECCV2022). As shown in Figure 1, this work proposes to first generate pseudo masks on the query image to mine query objects and then use the query object feature to generate the pseudo support (self-support) to match the query feature. The self-support of SSP performs the same operation to leverage the query prototype to match the query feature. The novelty and contribution of this work is extremely limited.

Questions

See the weakness.

Rating

2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

1 poor

Limitations

See the weakness.

Summary

This paper proposes a query-centric FSS method, which first performs rough segmentation based on the support features, then performs mask propagation based on the intra-semantic similarities, which is supervised by an adversarial learning process.

Strengths

1. The proposed method is technically sound. 2. The experiments validates the effectiveness of the proposed method. 3. The paper is easy to follow.

Weaknesses

1. The novelty is relatively limited. The idea of mask expansion based on the intra-semantic similarities is very similar to the SSP (Self-Support Few Shot segmentation): both methods first predicts a rough mask by identifying high-confidence matching regions and then refines the mask leveraging the rough mask as pseudo support features. Thus, in my view, the novelty of this paper is to supervise such refining process in an adversarial manner. 2. What is the physical rationale of the learned local proxies? There is no explicit explication of the physical rationale of learning the local proxies. In my view, the learnable local proxies acts similarly to the learnable queries in DETR model. In this sense, each proxy aims to capture some sort of local features in an object. I suggest presenting more theoretical analysis on the physical rationale of the learned local proxies. 3. Ablation study on the impact of the adversarial learning process, adversarial learning vs direct supervised learning by comparison between the real and fake local features.

Questions

1. More analysis of the physical rationale of the learned local proxies should be presented. 2. Ablation study on the impact of the adversarial learning process, adversarial learning vs direct supervised learning by comparison between the real and fake local features.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Check the weaknesses.

Summary

This paper studies few-shot segmentation (FSS). It proposes a new query-centric FSS model Adversarial Mining Transformer (AMFormer), which achieves accurate query image segmentation with only rough support guidance or even weak support labels. The core idea is to have a object mining transformer (G) that can achieve the expansion of incomplete region activated by support clue, and a detail mining transformer (D) to discriminate the detailed local difference between the expanded mask and the ground truth. The proposed method outperforms SOTA on PASCAL-5 and COCO-20 benchmark.

Strengths

+ The proposed method records strong performance over SOTAs.

Weaknesses

This is not a weakness but a question arises from not fully understanding the exp setting: 1. When you evaluate and compare against SOTAs in the experiments presented in Table 2 and 3, are the annotations, i.e. the masks used in training the the few shot learning, the same across all methods?

Questions

see above

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

This paper solves the problem of few shot segmentation. It differs from the existing view of solving this problem which leverages heavily on the exploration of the support samples. The core idea is to shift the framework from the support-centric to the query-centric. The problem method contains an object mining transformer and detail mining transformer. Through adversarial training, the method supress existing methods on two benchmarks.

Strengths

+: The idea is new for few shot segmentation. Paying more attention to the query has been less inverstigated in the literature. The paper demonstrates the benefit of query-centric solution. +: The results are encouraging. +: The writing is fluent and easy to follow.

Weaknesses

Table 1 is important for rethinking about the existing supprt-centric methods, more details about how the experimental detials to obtain Table 1 are necessary to include in the paper. How about the convergence of the adversarial training? Does the discriminative region localization module only deal with one class in the support label? How about the support image has more than one segmentation classes? It is unclear why should we have to use the specific architectures in the proposed G and D, is there any evidence or insight for the designs of such networks? The method achieves SOTA on most cases, not all the settings.

Questions

The paper is overall sound. Please see the weaknesses for questions.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper discusses the impact and limitations in the paper.

Summary

In this paper, the authors introduce a concept for solving the few-shot segmentation problem using weakly supervised learning concept. The method relies on weak support labels like scribbles or bounding boxes that highlight the object regions in the query. The segmentation is achieved through two blocks - the object mining transformer and detail mining transformer - during training, while only the object mining transformer is needed during inference. The proposed approach outperforms other methods, as demonstrated by the results and ablation studies. The writing is clear and helps readers understand the proposed concept effectively.

Strengths

The authors have developed a solution that can solve the few-shot segmentation problem under more practical scenarios. This means that only partial annotation on the support images is required, making it a practical tool. Additionally, the query-centric concept can lessen the reliance on support images and overcome the differences between support and target images of the same class, making the solution more applicable. Although the detail mining transformer approach proposed with part-level adversarial training can slow down training, it is not necessary during inference. Lastly, the solution works well as AMFormer outperforms previous works significantly.

Weaknesses

1. It seems that the model complexity is quite high due to the large-scale backbone, transformer-based object mining model, and multi-scale structure. 2. I am unsure about the task division among the proxies, and it seems that the number of required proxies might depend on the complexity of the data. I hope that the design of this part can be made more clear and understandable so that it would be easier to debug in case of any issues.

Questions

1. Incorporating adversarial training may affect the convergence and stability of the learning process. Can you provide more information on this? 2. Do you notice any signs of over-fitting in the suggested method? 3. How does the model complexity compare to the competing approaches in Table 2 and Table 3?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Have you considered how well the query-centric method will perform when dealing with small targets, like in satellite images? Specifically, I am wondering about its generalizability for different datasets, ex: satellite images or medical images.