Open-vocabulary semantic segmentation is a challenging task that requires segmenting novel object categories at inference time. Recent studies have explored vision-language pre-training to handle this task, but suffer from unrealistic assumptions in practical scenarios, i.e., low-quality textual category names. For example, this paradigm assumes that new textual categories will be accurately and completely provided, and exist in lexicons during pre-training. However, exceptions often happen when encountering ambiguity for brief or incomplete names, new words that are not present in the pre-trained lexicons, and difficult-to-describe categories for users. To address these issues, this work proposes a novel attribute decomposition-aggregation framework, AttrSeg, inspired by human cognition in understanding new concepts. Specifically, in the decomposition stage, we decouple class names into diverse attribute descriptions to complement semantic contexts from multiple perspectives. Two attribute construction strategies are designed: using large language models for common categories, and involving manually labeling for human-invented categories. In the aggregation stage, we group diverse attributes into an integrated global description, to form a discriminative classifier that distinguishes the target object from others. One hierarchical aggregation architecture is further proposed to achieve multi-level aggregations, leveraging the meticulously designed clustering module. The final results are obtained by computing the similarity between aggregated attributes and images embeddings. To evaluate the effectiveness, we annotate three types of datasets with attribute descriptions, and conduct extensive experiments and ablation studies. The results show the superior performance of attribute decomposition-aggregation.
Paper
Similar papers
Peer review
Summary
The paper proposes a decomposition-aggregation framework to address the problem of open-vocabulary semantic segmentation. The framework consists of a decomposition stage, where pre-trained language models (LLMs) such as ChatGPT are used to extract textual attributes for class names, and an aggregation stage, where a hierarchical structure is employed to progressively aggregate attribute features. The experimental results demonstrate the effectiveness of the proposed method.
Strengths
1. The idea behind the framework is straightforward and easy to understand. 2. The experimental results provide evidence that the proposed method is effective.
Weaknesses
1. The performance of the framework heavily relies on the quality and accuracy of the selected attributes. Incorrect or inaccurate attributes may degrade the segmentation performance. It would be beneficial to explore ways to regulate the risk introduced during the decomposition stage, both during training and inference. 2. While leveraging additional attributes from external knowledge improves the performance of open-vocabulary semantic segmentation (OVSS), it is important to consider whether explicitly decomposing class names into attributes align with the long-term goals of OVSS. Implicitly learning the "decomposition and composition" process might be a more desirable solution in the future.
Questions
When decomposing a class name into a set of attributes using a language model like ChatGPT, is it correct to say that the attributes generated by the language model are based on common sense rather than visual inspection, considering that the language model does not have access to the images?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
Refer to Weaknesses
Summary
[Task] This paper presents a novel decomposition-aggregation framework for open-vocabulary semantic segmentation, which addresses the challenges of segmenting novel object categories during inference. The existing vision-language pre-training approaches make unrealistic assumptions about low-quality textual category names, such as the availability of accurate and complete category descriptions in pre-trained lexicons. This work aims to overcome these limitations and proposes a framework inspired by human cognition for understanding new concepts. [Method] The framework consists of two stages: decomposition and aggregation. In the decomposition stage, class names are decoupled into diverse attribute descriptions to enrich semantic contexts. Two attribute construction strategies are employed: leveraging large language models for common categories and manual labeling for human-invented categories. In the aggregation stage, diverse attributes are grouped into an integrated global description, forming a discriminative classifier that distinguishes the target object from others. A hierarchical aggregation technique is introduced to achieve multi-level alignment and deep fusion between vision and text. The final segmentation result is obtained by computing the embedding similarity between aggregated attributes and images. [Experiments] To evaluate the effectiveness of the proposed approach, three datasets with attribute descriptions are annotated, and extensive experiments and ablation studies are conducted. The results demonstrate the superior performance of the attribute decomposition-aggregation framework. The paper concludes by mentioning the release of datasets and codes upon publication, which will enable further exploration and validation of the proposed method.
Strengths
[Novel Framework]: The paper introduces a novel decomposition-aggregation framework that addresses the challenges of open-vocabulary semantic segmentation. By drawing inspiration from cognitive psychology, the framework leverages diverse attribute descriptions to enhance category comprehension and improve segmentation performance. [Practical Relevance]: The paper highlights the limitations of existing vision-language pre-training approaches in real-world scenarios and addresses issues such as ambiguity, neologisms, and unnameability, which are critical for achieving accurate and comprehensive open-vocabulary semantic segmentation results. [Construction Strategies]: The paper proposes two construction strategies for generating attribute descriptions, utilizing language models and manual labeling. This provides flexibility and accommodates different scenarios, including both common categories and situations involving novel or difficult-to-describe categories. The inclusion of a hierarchical aggregation architecture enables the exploitation of potential attribute hierarchy, which can enhance the segmentation process and improve the quality of the final segmentation result. [Experimental Evaluation]: The paper evaluates the proposed framework on multiple datasets, demonstrating its superior performance over various baselines and competitors. The inclusion of thorough ablation studies further enhances the understanding of each component's contribution.
Weaknesses
[Weak baselines] The paper primarily compares the proposed work to LSeg, which may be considered a weak baseline as it is not the latest work in the field of open-vocabulary semantic segmentation. The achieved performance gains with the proposed framework, using RN101 as a backbone, are relatively modest at around 3% (3% is modest since the baseline is LSeg rather than the current SOTA work in open-vocab semantic segmentation). Given the complexity of the framework and training pipeline, it raises questions about the overall effectiveness of the proposed approach. Can you compare your work to some latest works as well? [Performance evaluation with class names alone] The experimental results presented in Tables 2 and 3 demonstrate that the performance on the custom datasets Pascal-5 and COCO-20 is only comparable to LSeg (LSeg is evaluated based on class names). While the capability of evaluating the model's performance without relying on class names is intriguing, it is more common and practically useful to support evaluation with class names. Therefore, could you please clarify whether the proposed approach's performance would be influenced when class names are provided as part of the inputs?
Questions
My main questions are listed in the weakness sections. I have a few more questions on the ablation study and model evaluation as follows: In addition to evaluating with exact class names, it would be beneficial to know if the proposed approach can handle hybrid inputs. For instance, can it work with rough class names such as "dog" without specifying the breed (e.g., "corgi"), combined with additional textual descriptions? Supporting hybrid inputs would enhance the flexibility and practicality of the proposed framework, enabling more realistic scenarios where users might provide partial or general class names along with descriptions. Addressing these concerns would provide a better understanding of the comparative performance of the proposed framework and its compatibility with evaluating segmentation results using hybrid inputs.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Yes, the authors adequately addressed the limitations.
Summary
Previous work does not address the issue of ambiguity or potentially incomplete textual categories and directly incorporates them into vision-language models without any post-processing. This approach hinders the text encoder's ability to fully comprehend the category. To overcome this limitation and provide comprehensive categorical information, the author suggests decomposing the category into more descriptive texts, such as attributes, using large language models (or human-designed prompts). Subsequently, the decomposed texts are aggregated using the proposed method.
Strengths
1. Sound motivation. The core idea of distinguishing categories by decomposing them into attributes and aggregating them sounds promising. 2. Suggesting a new dataset to support the motivation.
Weaknesses
1. Missing citations. The authors should include citations to recent open-vocabulary semantic segmentation literature and make a comparison with them [2, 3, 4]. 2. Although the motivation is good, the experiment is insufficient. The benchmark conducted by the authors is not sufficient to provide a complete comparison with recent open-vocabulary segmentation works. The authors conducted experiments on benchmarks widely used in few-shot segmentation literature, rather than in open-vocabulary semantic segmentation. To the best of my knowledge, only two papers have utilized this benchmark in an open-vocabulary setup (LSeg, Fusioner [1]). This choice prevents a comparison with recent state-of-the-art open-vocabulary methods such as Zegformer [2], OVSeg [3], or CAT-Seg [4]. 3. Lack of analysis of the motivation for **decomposition**. The provided analyses primarily focus on demonstrating the effectiveness of the aggregation modules, rather than specifically proving the effectiveness of the decomposition approach. It would be beneficial for the author to include ablation studies and analyses of the **decomposition** within the main paper.
Questions
The authors chose to aggregate visual and attribute information together in one model. However, considering the pre-training nature of vision-language models, the most naive way to achieve the same goal would be to concatenate all the attributes into a sequence and input them into a text encoder. The resultant text embedding can then be used as a classifier, which can be applied to existing methods such as Zegformer or OVSeg. I'm curious about the performance of this naive method, which is the most easiest way to confirm the validity of the motivation of decomposition.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The motivation seems promising; however, I think this motivation can be applied to all of the existing methods in open-vocabulary literature. In other words, the method the author suggests does not seem to emphasize the advantage of the motivation but rather has high similarity to previous work (Fusioner [1]). This paper would be much stronger if the author provided an analysis regarding the motivation instead of focusing solely on the technical design. [1] Ma et. al., "Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models", BMVC'22 [2] Ding et. al., "Decoupling Zero-Shot Semantic Segmentation", CVPR'22 [3] Liang et. al., "Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP", CVPR'23 [4] Cho et. al, "CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation", arXiv'23
Summary
This paper proposes a framework for decomposing class names into attributes for open vocabulary semantic segmentation. The authors obtain attributes from LLMs or hand-curate for unheard-of classes, then aggregates these attributes for obtaining a general descriptor for each classes. Moreover, the authors collect a dataset to demonstrate their effectiveness on addressing neologisms and unnameability for OVSS.
Strengths
1. The motivation for the paper, addressing the problems existing in pre-trained VLM-based paradigm for OVSS seems solid. 2. Evaluating the proposed method on a newly collected dataset, consisted of imaginary creatures, is an interesting method for evaluating how models can handle "neologism" and "unnameability" addressed in the paper.
Weaknesses
1. Insufficient evidence for supporting the work - The authors evaluate with COCO-20i[19] and PASCAL-5i[10, 13], which have brief, but well-known classes. These classes barely seem to suffer problems such as neologism or unnameability, and does not illustrate cases where ambiguity might be an issue. - Moreover, the collected attributes for COCO and PASCAL are not present in the paper or the supplementary materials, making it harder to validate how these attributes may contribute to the quality of the output segments. 2. Weaknesses in quantitative comparison and ablations - The current comparison with LSeg[17] does not seem to sufficiently provide evidence for the core contribution of the work of attribute decomposition-aggregation, as LSeg[17] does not improve with provided attributes. The improvements may seem to come form the fusion layers, which seems to beg similarity with Fusioner[21]. Comparison with Fusioner[21] would better illustrate the significance of this work, as it would be able to solely ablate the proposed attribute aggregation. 3. Flaws for addressing the presented problem - If the class names are ambiguous by their own, how are LLMs capable of distinguishing between this ambiguity? For example, how would LLMs be able to distinguish between the bird "crane" and the machine "crane" with the following prompt "Describe what a {crane} looks like in the image.", without further elaborating the context? - The claim that "segmentation through attributes is a more challenging task"(L.281) seems to be a bold claim, as the authors obtain the attributes as additional information from the class names, with the help from LLMs or human annotation.
Questions
1. The collected dataset, "Fantastic Beasts" is collected from a movie series, which may raise concerns regarding copyright issues. Can the authors include discussions, or clarify how the copyright may not be of an issue?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Limitations regarding the collection of the dataset should be further discussed.
I thank the authors for the detailed response. Additional experiments does answer a lot of my concerns, and I think they should definitely be in the paper. I also agree that COCO/PASCAL are indeed standard benchmarks. I have adjusted my initial rating, but my remaining concerns are: 1. Since "textual ambiguity" requires human annotation, this seems like it can be easily handled by other methods by just specifying "crane bird" or "crane machine". How can the proposed method be more effective or efficient in this perspective? 2. Comparison with SOTA works. From current results, the authors use "direct aggregation" when evaluating with SOTA works, where all the attributes are concatenated to a single sentence for CLIP. The problem is that CLIP can struggle with this long, detailed description and might better perform with short, distinctive descriptions, hence current evaluation seems a bit unfair. Since the use of human annotation seems inevitable, can the authors evaluate SOTA works on Fantastic beasts with a **short description**, for example like "small black creature"? 3. Further investigation of "cls+attr". I think the results from Table 6 are fascinating, and can be joined with Table 7. Since a more common case would be having the class name, "cls+attr" seems to be a more reasonable and general case compared to only having attributes. In this perspective, can the authors show results when having **wrong classnames**, where the attributes can act as a **fallback** for addressing neologism or unnameability? Thanks.
We appreciate the reviewer for acknowledging our rebuttal, and we are pleased that our feedback addresses most of your concerns. We will definitely include all these additional experiments in the revised paper. Furthermore, we respect the responsibility of the reviewer, in providing constructive suggestions that can enhance the paper and make it even better. Here is our response to the remaining concerns: > **1. Other methods can handle this by specifying "crane bird" or "crane machine". How can the proposed method be more effective or efficient in this perspective?** We agree that other methods can handle this scenario by inputting "crane bird" or "crane machine". However, it is worth noting that "bird" and "machine" are also one of the attributes associated with concepts of a "crane". Specifying "bird" or "machine" is considered as specifying crane's attributes. These instances further align with the idea we have proposed, i.e., decomposing into attributes and aggregation, serving as compelling examples that highlight the generality of our motivation. Besides, as stated in the response to the third question of R-mZMs, our method is capable of handling "hybrid inputs". For example, it can process general class names (e.g., "dog") or specify the breed (e.g., "corgi"). This can enhance the effectiveness and efficiency of our method. We can consider "crane bird/machine" to be an example that falls into this situation. As demonstrated in Table 6, our method outperforms existing methods when using hybrid inputs. > **2. CLIP can struggle with long, detailed description when concatenated, hence current evaluation seems a bit unfair. Can the authors evaluate SOTA works on Fantastic beasts with a short description?** We first clarify that after concatenation of all attributes belong to their class, none of the categories have exceeded the specified maximum sequence length for the model to process. We present statistical data, with an average length of approximately 55 for all categories, and the longest length is 75. These lengths are all well within the maximum sequence length of 77 (the CLIP model we used). So we believe that the length of the concatenated attributes is not a problem for CLIP to handle, thus the current evaluation is fair. As suggested, we also conducted experiments on Fantastic Beasts with a short attributes description. In order to ensure maximum attributes diversity such as color, shape, parts, etc., we selected *3* most distinctive attributes for each category in Fantastic Beasts (3 is significantly fewer compared to the original 15 attributes). After concatenation, the total length of these 3 attributes is not exceed 9, with an average around 6 (some attributes may contain 2 words, for example, "small, blue body, thin wings"). The results are shown in Table 8 below. Our methods still demonstrate the superiority. *Table 8: Evaluation on Fantastic Beasts with a short attributes description.* Model|Backbone|Training data|mIoU(short attr) :-:|:-:|:-:|:-: LSeg|RN101|PASCAL VOC-15|40.7 Fusioner|RN101|PASCAL VOC-15|41.1 Zegformer|RN101|COCO-Stuff|52.4 OVSeg|RN101|COCO-Stuff|54.0 CAT-Seg|RN101|COCO-Stuff|54.6 **Ours**|RN101|PASCAL VOC-15|47.4 **Ours**|RN101|COCO-Stuff|**56.0** > **3. "cls+attr" seems to be a more reasonable and general case compared to only having attributes. In this perspective, can the authors show results when having wrong classnames, where the attributes can act as a fallback?** We agree with reviewer's suggestion where the attributes act as a fallback when having wrong class names. In real world scenarios, such situations can potentially occur where users may input incorrect class names while also providing accurate attributes. Considering this, we give experiments based on Table 6 and Table 7, and treat the wrong class name as one incorrect attribute. The wrong class name is randomly selected from the remaining categories, excluding the correct category. As shown in Table 9, inputting wrong class names will lead to extremely poor results. Since class names hold core information (mentioned in the general response), the wrong class name will result in a completely incorrect textual classifier for VLP-based methods. On the contrary, however, the attribute can provide correct information from multiple distinct or complementary perspectives, which can act as a "fallback" when the class name is wrong. *Table 9: Performance of wrong class name inputs, where attributes can act as a fallback.* Model|Backbone|Training data|mIoU(correct cls)|mIoU(wrong cls)|mIoU(wrong cls+attr)|$\Delta$w/ correct cls :-:|:-:|:-:|:-:|:-:|:-:|:-: LSeg|RN101|PASCAL VOC-15|47.4|<10|41.6|12.2% Fusioner|RN101|PASCAL VOC-15|46.4|<10|41.8|9.91% Zegformer|RN101|COCO-Stuff|86.2|<10|76.1|11.7% OVSeg|RN101|COCO-Stuff|92.6|<10|81.3|12.2% CAT-Seg|RN101|COCO-Stuff|93.7|<10|84.7|9.6% **Ours**|RN101|PASCAL VOC-15|50.0|<10|45.9|8.2% **Ours**|RN101|COCO-Stuff|93.3|<10|86.0|**7.82%**
Thank you for the additional response. I am glad to see my remaining concerns are sufficiently addressed in the provided results, and based on them, I am convinced that this paper holds interesting values for the field of open-vocabulary segmentation. I am looking forward to see the paper strengthened with the additional experiments, and hence adjust my rating accordingly.
Thank you for your detailed review and taking time to provide feedback on our paper! We are glad to hear that our additional response has sufficiently addressed all your concerns. We are excited to continue our work on open-vocabulary segmentation and will incorporate the additional experiments to further strengthen our paper. Thank you again for your valuable comments.
Thank you for your response! The experiments all look really interesting, and I believe the results should be included in the main paper. I hope to see how future work will further develop this idea.
Thank you for the response and appreciation! All experiments and results will certainly be appended in the revised paper. We also value your suggestion. Considering the value of attributes, some possible directions for future work include: 1. More comprehensive visual tasks. To extend our motivation and framework to various tasks, such as instance segmentation, object detection, and image retrieval, etc. 2. Robust strategies on attribute noise. To design more efficient and effective attribute decomposition and aggregation methods that can handle large-scale and noisy attribute descriptions, such as using attention mechanisms or graph neural networks. 3. Hierarchical cross-modal alignment. To investigate the hierarchical structure within attributes, as well as aligning them with different levels of visual features. A potential approach could involve leveraging hyperbolic geometry, by embedding them in a hyperbolic space. In conclusion, these future works aim to guide the community towards more practical directions. We hope these advancements will better serve and enhance our daily lives.
I thank the authors for the detailed response. I have also read through all reviews given by other reviewers and the author's corresponding responses. The paper looks more interesting and complete if all the additional experimental results are added. Even though the method for this work is straightforward, I do believe that the value and inspiration this work will bring to the community.
Thanks for the support to our paper! All the additional experimental results will definitely be added in our revised paper, and thank you again for considering our responses.
I appreciate the authors for their efforts in addressing my inquiries. The majority of my concerns have been satisfactorily resolved, leading me to recommend the acceptance of this paper. Regarding the authors' explanation for the perception of "modest improvement" in their method, as indicated by the quote "Most of the latest works are trained on a larger dataset COCO-Stuff with more classes and more images," I do urge the authors to consider conducting experiments on a larger dataset and with a larger model. This exploration could shed light on the impact of both data size and model scale, which I think intriguing.
We appreciate your support for our work and the positive comments. We are glad that our feedback addressed your concerns. Here is our response to the remaining concerns: > **Experiments on a larger dataset and with a larger model.** We thank your suggestion and agree that this exploration could provide interesting insights into the impact of both data size and model scale. As suggested, we conducted experiments on a larger dataset (COCO-Stuff) and with a larger model (ViT-L), following the recent open-vocabulary literature. PASCAL VOC contains 20 categories with total 1464 images for training, and PASCAL VOC-15 refers to its 4-fold cross-validation (15 categories for training). COCO-Stuff contains 171 categories with total 118k images for training, which is greatly larger than PASCAL VOC. For backbones, we also use ViT-L, which is a larger model than RN101. The results are shown in Table 10 below. Using a larger dataset and a larger model can indeed improve performance. In particular, when the dataset is larger, the improvement can be more significant. *Table 10: Performance of PAS-20 on a larger dataset and a larger model.* | | PASCAL VOC-15 | COCO-Stuff | $\Delta$mIoU(Dataset) | | :---------------------: | :-----------: | :--------: | :-------------------: | | RN101 | 49.1 | 91.6 | 42.5 | | ViT-L | 56.4 | **94.1** | 37.7 | | **$\Delta$mIoU(Model)** | 7.3 | 2.5 | - |
Decision
Accept (poster)