Renovating Names in Open-Vocabulary Segmentation Benchmarks

Summary

Open-vocabulary models use class names as text prompts for unseen categories’ generalization during training. In this paper, the issue of imprecise or even wrong class names from existing datasets is specially studied. One simple and general framework is proposed for automatic dataset renaming, that is, first narrow down the whole name search space to a curated list of candidate names, by using original class names with contextually relevant nouns from image captioning; next employ a specially trained renaming model to select the best-matching candidate name for every ground-truth segmentation mask. To demonstrate the effectiveness, two types of practical applications are explored, i.e. using renovated names to train open-vocabulary models, and applying renovated names to improve evaluation of open-vocabulary segmentation.

Strengths

[+] Due to the fact that class names are often directly used as text prompts, they have a significant impact on the performance of open-vocabulary tasks. Therefore, focusing on the noisy issue of category names is on the right path. [+] Leveraging foundation models to automate the renaming process indeed reduces the manual labor. [+] The paper is easy to follow and understand, having clear logic. [+] Some experiments are conducted to demonstrate that renovate names help to train models with stronger open-vocabulary capabilities and refine evaluation benchmarks.

Weaknesses

[-] Noise type & noise rate. The authors partially visualize the noise issue of some class names, such as inaccurate or too general or lack contexts, which is worth encouraging. But, are there only these three types of noise in real-world scenarios? What are the noise rates for different datasets? And is the effectiveness of RENOVATE the same for different types/ratios of noises? Thorough statistics are needed to clarify the performance bounds. [-] How many ground-truth masks are needed when sorting candidate names using visual context? How to ensure consistency in visual alignment, as visual masks typically have significant intra-class variances (such as humans having different hairstyles)? [-] To obtain candidate names, GPT-4 and image captioning are used. For GPT4, it is usually sensitive to prompts, how to ensure the quality of generating candidate names (Quantitative comparisons are needed)? Do you need additional labor to perform a double-check after combining results from GPT-4 and captioning? [-] Since renovating names for open-vocabulary understanding seems to be one general idea, why only focus on segmentation tasks? Will the conclusion of this paper be exactly the same for classification or detection? For example, when using visual samples to match the best candidate, the intra-class visual variances of classification are definitely greater than those of segmentation.

Questions

[-] In the related-work section, some citations are missing, which may mislead readers/reviewers. [1,2] have explored the text issue of class names in open-vocabulary segmentation, with some overlapping with this paper. The reviewer encourages including them and conducting detailed discussions and comparisons. [1] AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation. NeurIPS 2023 [2] Multi-Modal Prototypes for Open-Set Semantic Segmentation. Spring IJCV

Rating

6

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

As the authors said, RENOVATE could inadvertently propagate biases from the foundational models into the new names. To mitigate potential negative societal impacts, they advocate for verification of names in critical applications. Also, the authors acknowledge that, the exploration remains incomplete, and will further make refinement to explore more model backbones and scale up to large-scale datasets.

Summary

The paper explores modifying the class names associated with mask annotations in the segmentation datasets in the context of open-vocabulary segmentation. A method, RENOVATE, is described to perform such renaming in an automated pipeline. RENOVATE leverages an image captioning model to generate contextual words for each class. GPT-4 is prompted by class name and the contextual world to generate a pool of candidates. A segmentation model is trained to select among the candidate names using the mIoU as a proxy metric. The refined class names are then used to train open-vocabulary models, which show improvement in performance.

Strengths

1) The paper explores an interesting angle of open-vocabulary segmentation: improving the names assigned to each segment. This is a novel study, highlighting the value of precise and varied captions for downstream training. 2) The writing of the paper is engaging. Particularly, the ample use of examples helps illustrate key points around the proper use of terms. 3) The paper conducts a very small scale study (n=20) human study showing a preference for new names. 4) The conducted ablation study verifies some components of the pipeline.

Weaknesses

1) The section on the training of the ranking model to select among the candidate names could be clearer and more streamlined. It is not particularly clear whether a model is trained for each category or if a single model is trained for all categories. - If it is the former, then there are certainly some computational concerns and limitations that the method imposes. It would be useful to discuss the computation cost wrt to the number of classes in the data. - If it is the latter, then the ability to use or re-use this trained model for the downstream task of OV segmentation could be considered. It seems that the effect of gt-mask attn bias is limited (B.2) as randomisation performs better. Also, reporting the performance of this model on the downstream task is of interest. 2) In the same section, the design and presentation of a specialised architecture for the renaming could be better motivated. Since a proxy of a segmentation task is employed, is the new architecture even necessary? Could existing model architecture perform this task? Why do existing OV segmentation model pipelines not meet the requirements of the proposal's renaming stage? 3) The evaluation of the study for the improved performance of the segmentation models is limited -- only a single architecture is explored. Other architectures would help rule out particular drawbacks of the FC-CLIP being addressed with renaming and would provide more evidence to the broader conclusion about renaming helping model training (L310). 4) While the new names might help identify particular problems in models (L297), the reasoning behind renovating names in evaluation is rather backwards. While some misclassifications might be due to the similarity in labels, such as those identified as "benign" by the authors, they are still misclassifications as defined by the task. This then creates a problem in downstream use. For example, two masses in an X-ray might appear very similar due to both being a collection of cells, but their placement and small characteristic outline patterns will determine the difference between benign and dangerous growths. Thus, one should seek to highlight key errors in the evaluation instead, even if the labels are close. It seems that a different phrasing of the conclusions (L296): the learned semantics are not sufficient to differentiate between semantically adjacent classes. 5) [Minor] Finally, the proposed pipeline itself might be less applicable outside of the academic where established datasets with limited label sets are used. Would gathering a dataset with varied captions from the onset make the RENOVATE method redundant?

Questions

- It would be good if the issue surrounding the training of the renaming model (W1) could be clarified in the rebuttal. - Additionally, providing some commentary on the weaknesses (W2) and (W4) would help improve the interpretation of the paper. - Furthermore, it would be good to discuss future applicability of the pipeline, or whether the better path is to just collect more precise captions for training data. Overall, the paper is well-written and presents some insights into the treatment of data for open-vocabulary segmentation. There are some limitations in the evaluation. I currently rate the paper as BA prior to rebuttal.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper touches on the potential to propagate existing biases in the foundational models it employs, which is a big issue, and it is important that it is mentioned. I would recommend conducting an ethics review or obtaining IRB approval for the human study, even if it is not strictly necessary according to the respective institutional guidelines.

Summary

The paper considers the problem of open-vocabulary segmentation. The task requires segmentation models to recognize categories outside of the training taxonomy. This usually means learning a joint image-text feature space and classification via similarity matching between class text embeddings and segmentation mask embeddings. The paper aims to improve the suboptimal text descriptions of the dataset classes, which are often too general, ambigous and context dependant. The proposed method describes a strategy for automated renaming of the assigned class descriptions at the segment level. This means that each ground truth segment gets assigned with one or more text descriptions that describe that particular segment more specifically. Training with such enriched text descriptions improves the model performance on both the source and target datasets. The experimental study follows the usual setup where COCO is used as source dataset, and ADE20k and Cityscapes as target datasets.

Strengths

The paper considers an important and underexplored problem in open-vocabulary segmentation. To the best of my knowledge, OpenSeg [5] is the only prior work that also considers the problem of suboptimal class descriptions and corresponding renaming solutions. The paper provides a proper comparison and discussion of the differences. The proposed renaming method is fully automated, which represents a significant step forward w.r.t. to the related work that requires manual inspection. The described approach is technically sound and properly explained. The proposed method seems dataset and model agnostic. This means that it should easily generalize to other datasets and would be beneficial for training of different open-vocabulary segmentation models. The authors promise to share the code and the renamings for popular datasets which would have positive affect on future research in open-vocabulary segmentation. The results reveal that training with renovated vocabulary leads to increased segmentation performance across different metrics and datasets. Furthermore, including the renovated vocabulary only in the evaluation phase leads to increased performance of the open-vocabulary models pretrained with regular datasets.

Weaknesses

Some parts of the paper remain unclear. For example, I am confused about the batch collation for training the renaming model. Figure 2 suggests that the transformer decoder in a single forward pass receives candidate names from a single ground truth class. Does this mean that for a single image the number of forward passes through the transformer decoder is equal to the number of ground truth segments? Would you say then that a single training example is consisted of an image, a single ground truth segment and multiple candidate names? Is then a batch size equal to the total number of segments in the sampled images? Does the batch size 16 from line 239 refers to the number of images or number of segments? Does this type of training affects the memory requirements significantly? Are there any redundancy in terms of the tensor representations that have to be kept in the memory in order to enable parallel training on multiple segments? I suggest authors to clarify these questions. I also dont understand the details of using renovated names during the evaluation. Does this mean that all the candidate names are just appended to other naming variants of that particular class? It seems to me that we do not need to train the renaming model if we would like to use the renovated names just for the evaluation. Is this correct? How many naming variants per class do you get on average when you concatenate the proposals from the original set, the OpenSeg and the renovated names? Does this affect the inference speed significantly? If you also use the usual templates (a picture of *class name*, an image of *class name*, etc.) it seems to me that there might be a lot of different textual representations for a single class. How does this inference really work? The renaming model should be compared to some stronger baseline which also has the ability of candidate selection. For example, you could assign each segment with a candidate name that achieves highest similarity with the segment representation based on the CLIP embeddings. The segment representation could be average-pooled feature representation from the CLIP visual encoder. I question the necessity of training the mask prediction module just for renovating the class name associated with some segment, if we already know the corresponding ground truth mask. Could you please comment on this issue and maybe discuss some possible alternatives? In Table 3. the results obtained with OpenSeg names should correspond to the original FCClip results. However, in some cases the reported performance is lower than the FCClip performance reported in the official repository. Why is this the case? As shown in Table 3., training with renovated names leads to marginal gain in some cases (e.g. PQ on Cityscapes). I am not sure if the x-axis on figure 5 shows the number of images available in the training dataset, or the number of iterations trained on the full dataset? If these experiments consider subsampled training subsets, you should explain how the training images were selected. I am a bit surprised about the effectiveness of the negative sampling. What would be the effect of including additional renovated names of the ground truth class as negatives? This reminds of hard example mining, as it would force the model to differentiate between similar semantic concepts.

Questions

Please consider the questions from the weaknesses section that would clarify training of the renaming model and the evaluation procedure with the renovated names. Also, please discuss the possibility of evaluating the renaming model with another baseline with candidate name selection ability. Please find the suggestion in the weaknesses section, which considers a baseline based on pooling of CLIP features.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors discussed the limitations in the section 6. The main concern is regarding the bias transfer from the foundation models.

Thank you for the rebutall

I wanted to acknowledge and thank the authors for their detailed responses. I believe the majority of my concerns have been addressed, and I support the proposed changes. I will update my recommendation accordingly.

Thanks for your review

We thank the reviewer for the ethics review. As the reviewer mentioned, in our paper, we have acknowledged that bias propagation from foundation models is indeed an issue and we advocated for verification of names in critical applications. In case the reviewer sees further issues, we would be happy to address them.

Thanks again to all the reviewers

As the discussion period concludes, we would like to express our sincere gratitude to all the reviewers for their valuable feedback and suggestions. We are pleased to see that all the reviewers acknowledged our rebuttal effectively addressed their concerns and subsequently raised their recommendation ratings, recognizing the value of our paper. We will carefully consider all the feedback and incorporate it into our revised paper.