Understanding Multi-Granularity for Open-Vocabulary Part Segmentation

Summary

This paper proposed PartCLIPSeg for open-vocabulary part segmentation. This framework leverages generalized parts and object-level contexts for generalization in fine-grained parts, integrating competitive part relationships and attention control techniques. Extensive experiments on three datasets demonstrate the superior performance of PartCLIPSeg.

Strengths

1) The paper is well-organized, good writing and easy to understand. 2) The topic of open-vocabulary part segmentation is meaningful and underexplored. 3) Significant improvement compared to the baseline methods.

Weaknesses

1) Figure 3 (overall architecture) can be refined to clearly emphasize the contribution. 2) The analysis and explanation of the experiment are not sufficient. Although the method achieves a significant improvement on unseen categories, the results of seen categories on the Pred-All setting of the three datasets show different trends. Specifically, the results of the seen categories show a decrease compared to the baseline in Tables 1 and 3 (There exists an annotation mistake (41.70).), and a significant improvement in Table 2. Please discuss this result. 3) Ablations of loss functions. We notice that adding some loss function in some cases may lead to suboptimal results and show different trends on different datasets. For example, the second column of the Pred-All setting in Table 5. Please discuss this difference.

Questions

How is the threshold γ in Equation 7 determined? How does this threshold affect the results?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The unseen categories of the three datasets in the experiment may have been seen during the CLIP pre-training, and we want to explore how well PartCLIPSeg generalizes to the “truly” unseen categories that have not been seen in the pre-training or from other fields

Summary

This paper proposes PartCLIPSeg, which builds upon the CLIPSeg model and extends it to handle the unique challenges of part segmentation. This paper introduces several components to address the challenges in OVPS. 1. Generalized parts with object-level contexts: This approach combines generalized part information with object-level context to mitigate the lack of generalization in fine-grained parts. 2. Attention control: This method introduces competitive part relationships and attention control techniques to minimize overlap between predicted parts and enhance activation for underrepresented parts. This mechanism is a new addition to the field. The paper conducts experiments on OVPS’s baseline datasets. The result shows that PartCLIPSeg outperforms existing SOTA methods in most settings. The Ablation primarily focuses on the impact of the proposed attention control loss, which lacks a comprehensive analysis of other key components of this paper.

Strengths

- Introducing generalized parts with object-level contexts: The paper proposes an approach that combines generalized part information with object-level context to enhance the model’s ability to generalize to unseen object-part combinations. - Developing an attention control mechanism: The paper introduces an attention control mechanism which helps to address the issues of ambiguous part boundaries and missing small or infrequent parts. - Promising result on baseline benchmark: The paper demonstrates state-of-the-art performance in most settings of OVPS.

Weaknesses

- Insufficient ablation studies: The ablation studies primarily focus on the proposed attention control loss, which is not sufficient to prove the effectiveness of the proposed method. 1) The impact of combining object-level and part-level pseudo-label is not explored. 2) The paper does not provide sufficient evidence that the proposed losses work as intended, beyond their impact on the final metric. Further discussion is needed to validate whether the losses effectively address the challenges they were designed to tackle. - Limited comparison with other OVPS methods: The paper compares PartCLIPSeg primarily with CLIPSeg and CATSeg, which are designed for open-vocabulary segmentation and not specifically for OVPS. The comparison with other OVPS methods is lacking. - Limited discussion on computational complexity: Since the paper introduces a new attention control mechanism, a detailed analysis of the computational requirements and speed trade off could be helpful for a more comprehensive understanding of the proposed method. Inadequate qualitative analysis: Insufficient exploration and insights of the model’s performance on challenging and failure cases.

Questions

See Weakness.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

No societal impact.

Summary

This paper identifies three key problems of current open-vocabulary part segmentation, namely, lack of generalization, ambiguous boundaries, and missing underrepresented parts. The paper then proceeds to propose solutions to fix these problems. Experimental results show that the proposed method outperforms previous state-of-the-art methods.

Strengths

This paper identified some key issues in current open-vocabulary part segmentation and proposed effective solutions respectively. The paper was easy to follow, with good figures to demonstrate key concepts. The experiments also demonstrate that the methods can lead to nontrivial gains.

Weaknesses

1. **Limited discussion of previous open-vocabulary part segmentation work and their relationship to this work.** For example, I quickly searched and read the VLPart paper [1], which seems to parse objects first, then parts, thus avoiding the object-level misclassification issue (or the "lack of generalization" as named by the authors). Indeed, the VLPart seems very natural to me. So I am actually surprised to find that there is such an issue (mixing objects and parts during classification) as indicated by this paper. In general, the related work section provides very limited introduction of the previous open-vocabulary part segmentation methods, esp. how they are related to the proposed approach. There are also **no descriptions of the compared baselines in the experiment section**.This made readers hard to make a fair assessment of the contribution of this paper. 2. **Limited ablation studies to understand the proposed approach.** Some key questions are unanswered: how much does each proposed component (object-level context, attention control) contribute to the performance gain? In what way or what types of data? It might also be helpful to compare the Pred-Obj performance because that helps disentangle the localization and classification ability of the approaches. Current comparison mixes many components together and it's difficult to gain insights about how the gain is achieved. 3. **Lack of addressing directly the identified key problems**. The paper starts with three key issues and then propose different ways to fix them. However, in the experiments, there are no direct addressing to these problems any more (except Table 6 for small parts). It would be great if the experiments can clearly demonstrate how the three problems are mitigated by the proposed approaches. 4. **Lack of discussion on the training details in the paper**. For readers who are not familiar with CLIPSeg, it is impossible to tell how the model is trained and what datasets are used for training. There is also **a lack of descriptions of the compared baselines in the paper**. Without these information, it's hard to understand the performance gains. Minor issues: 1. I found it confusing to call it "lack of generalization" when the dog's parts are misclassified as sheep's or cat's. Isn't it just misclassification, or object-level misclassification? 2. Texts on the figures are way too small, esp. in Figure 2 and Figure 4. 3. The "supplementary material" for this paper should be put as an appendix to the main paper file, instead of as a separate file.

Questions

1. What is "incomplete guidance from knowledge-based, multi-granularity characteristics of parts"? It is unclear from the texts. It seems like an important motivation for the design of the attention control. Some more analysis or discussion would help here. 2. In Table 1 and Table 3, why are the performance of PartCLIPSeg on "seen" classes so low? From the model design, I don't see a reason for this. Do the authors have an explanation? 3. How to understand the "failures" in Table 6 where PartCLIPSeg cannot improve the performance from the CLIPSeg (such as "bird's eye") or PartCLIPSeg also has very low performances (such as "sheep's eye", "cow's neck")? And are all these parts indeed *small*? For example, "cow's leg" doesn't seem to be small to me. How do the authors define what are small parts?

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors have addressed the limitation and potential societal impact of the work.

kicking off reviewer-author discussion

Thanks the authors for the rebuttal, and reviewers' for the comments. We are in the period of author-reviewer discussion. - Reviewers, please read through all the reviews and rebuttal and see if authors' responses have addressed your and others' important questions. - Authors, please follow up with any reviewers' further comments. Cheers, AC