Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation

Summary

This paper proposes that the traditional WOVSS approach lacks effective supervision of group tokens. In this paper, visual patch tokens are modeled through the GMM model to obtain prototype features that can be used to supervise each group token. Experimental results show that the proposed PGSeg achieves state-of-the-art performance on several benchmark datasets.

Strengths

1. Appling explicit supervision for each group token by constructing prototypes is novel. 2. The proposed PGSeg can reach good performance without large vision-language foundation models.

Weaknesses

Major weakness: 1. As shown in Table 3, The performance of the baseline (GroupViT) drops a lot when using CC12M+RedCaps compared to CC12M+YFCC14M(Setting in the original GroupViT). To demonstrate the effectiveness of PGSeg, the performance of PGSeg trained with CC12M+YFCC14M should be reported. 2. Prototype P is an important parameter of this article. If prototype P is not updated during training, how much impact will it have on PGSeg performance? 3. The proposed PGSeg heavily relies on the EM algorithm to estimate the parameters of GMM, which may makes the model hard to train. What is the training time cost of PGSeg compared to the baseline(GroupViT). Minor weakness: 1. Text in Figure 2 and Figure 3 are too small, which makes the workflow of PGSeg hard to read.

Questions

See weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

No limitations are discussed.

Summary

This paper presents a method for addressing the challenge of granularity inconsistency in weakly open-vocabulary semantic segmentation (WOVSS). To tackle this issue, the authors propose a technique called non-learnable prototypical regularization (NPR). NPR leverages explicit supervision from prototypical knowledge to guide group tokens, leading to improved object segmentation characterized by enhanced compactness and richness. Extensive experiments have been conducted to validate the effectiveness of this method on several benchmark datasets.

Strengths

- The authors' use of prototypical knowledge to tackle the performance bottleneck caused by granularity inconsistency is a well-motivated and intriguing idea. - The effectiveness of the proposed method is demonstrated through comprehensive experiments on multiple benchmark datasets. The results surpass those of previous works, establishing a new state-of-the-art performance.

Weaknesses

- The term "granularity inconsistency" is initially unclear in the paper, specifically in lines 37-40 and the caption of Fig.2. Although Sec.3 provides clarification, it would be preferable to have a clear explanation in the abstract or introduction, as it serves as the main motivation for the proposed method. Alternatively, improving the caption of Figure 2 would help better understanding. - In Sec.5, there is a lack of explanations for the experimental results. For instance, in the "Comparison with Zero-shot Baselines" section, there are insufficient explanations regarding why specific methods (e.g., ViT-8S) were chosen as baselines and the implications of the results beyond superior performance. To provide a more comprehensive analysis, the experiments section should include additional explanations elucidating the intended message conveyed by the results. - Conducting ablation studies to evaluate the benefits of each NPR strategy (T-NPR only or I-NPR only) in terms of compactness and richness would be valuable. This would offer a clearer understanding of the effectiveness of each module by confirming the respective impacts of the NPR strategies on compactness and richness. - (minor) In Fig.~4 (bottom), the airplane and contrail exhibit similar colors (purple and red), making it difficult to ascertain whether PGSeg is superior to GroupViT. To demonstrate the superiority of the proposed method, it would be helpful to differentiate the airplane and contrail more distinctly by modifying their colors or exploring alternative visual representation approaches.

Questions

Please see the weakness above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No, the authors did not discuss the limitations and potential negative social impact in the main text.

Summary

This paper studies the problem of weakly open-vocabulary semantic segmentation, which aims to segment objects of arbitrary classes using only image-text pairs. The paper proposes a prototypical guidance segmentation network, which uses non-learnable prototypes estimated from image and text features to regularize the group tokens that cluster the image patches. The paper shows that the proposed method improves the segmentation results with increased compactness and richness of the group tokens, and achieves state-of-the-art performance on several benchmark datasets.

Strengths

1. The paper introduces a novel concept of non-learnable prototypical regularization. The idea of the paper is reasonable and novel. 2. The motivation and the methods are illustrated clearly in Introduction. 3. The performances improvement on VOC and COCO are significant.

Weaknesses

1. Ablation studies are not enough. The number of prototypes should be more for ablation study. The affect of the position and number of PG-Unit should be explored in ablation studies. 2. In Table 3, PGSeg shows the superiority on two datasets VOC and COCO, but on Context, the performance of PGSeg is lower than pervious methods. 3. Fig.3 is not clear enough to illustrate the proposed method. The text in the picture is too small to read.

Questions

1. The comparison is not fair enough in Table 3 since the training data are different. Adding PGSeg(CC12M+YFCC14M) is fair to compare the proposed method and ViewCO and GroupViT. 2. Why the proposed method performs much better on VOC and COCO but much lower on Context? What is the uniqueness of this dataset? 3. Some previous methods that tackles the semantic segmentation task using GMM should be discussed. e.g., "GMMSeg: Gaussian Mixture Models for Deep Generative Semantic Segmentation"

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors did not addressed the limitations. I suggest to add more experiments and ablation studies.

Summary

The paper addresses the problem of weakly open-vocabulary semantic segmentation, which is to segment images of novel classes without any segmentation annotation. To segment unseen classes, a model is trained to align between image features and text caption to learn to associate between image patches and class names. The proposed work argues that there is an discrepancy between visual-textual alignment during training and pixel-wise grouping during inference. Thus, the paper proposes a non-learnable prototypical regularization (NPR), which leverages Gaussian Mixture Model to provide pixel-wise grouping supervision to better align the model for segmentation task. Experiments are conducted on PASCAL, COCO, ImageNet and LVIS datasets.

Strengths

+ The idea of direction of weakly open-vocabulary semantic segmentation is impactful since it can significantly reduce the annotation effort especially for the costly segmentation tasks. + The proposed direction of using Gaussian Mixture Model to group image patches into groups for additional training supervision is sensible and interesting especially this doesn't require any supervision. + The paper is self-contained and provides enough details for result reproduction.

Weaknesses

+ The reviewer finds that the emphasis on the non-learnable aspect of the algorithm is misleading. Specifically, the proposed method does use expectation-maximization to compute the prototype which is a form of learning algorithm, which contradicts with the claim about non-learnable prototype regularization. Moreover, the reviewer is not convinced on why "non-learnable" is a desirable feature for segmentation given given that learnable/optimizable components would make the model fully differentiable and can be easily trained with gradient based optimization. Can the paper clarify this? + The experimental results appears to be inconclusive. Although the paper shows that the proposed method can achieves strong improvement with limited number of training data (using only CC12M table 3), but when scaling to large dataset, the improvement diminishes to less than 1%. Can the paper provide justification on this? + It is quite surprising that the model seems to be insensitive to the number of groups used during training (as shown in Table 5) where the difference is less than 1%? The number of groups should strongly influence performances as too many groups could introduce training noise and too few would not capture the full information in image. Can the paper justify this?

Questions

Please refer to the weakness section

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Sufficiently addressed

A minor mistake in Q2

We are sorry for a mistake that the descriptions on the results reported in **Q2** are inadvertently reversed, where the modified version is as follows: |Method | w/ update | w/o update | |:------------------:|:------:|:------:| |PGSeg (RedCaps12M + CC12M) | 53.2 | 51.8 ( $\downarrow$ **1.4**) | |PGSeg ( CC12M)| 49.0 | 47.5 ( $\downarrow$ **1.5**) | We sincerely regret this mistake, and apologize the inconvenience this may have caused.

Follow-up Discussion

Dear all reviewers, We sincerely appreciate all reviewers' constructive comments on our submission, which actually helps us improve our submission. As Discussion Stage 2 is about to end, we highly appreciate knowing if our responses have addressed your concerns. If you have any questions, comments, or concerns left, please let us know, we would like to provide more discussion with the reviewers. Thank you very much again. Best, The authors of Submission