Hierarchical Open-vocabulary Universal Image Segmentation

Summary

In this paper, the authors target open-vocabulary setting and propose a universal framework for open-vocabulary semantic/instance/panoptic segmentation. The whole framework is DETR-like. And to deal with the discrepancies between the thing and stuff classes, the authors ultilize independent decoders for thing and stuff classes. The authors benchmark their method on various datasets and achieve remarkable performance.

Strengths

1) The idea is simple and easy to understand. 2) The results seem promising.

Weaknesses

1) The idea seems incremental. Compared to UNITEXT, little new knowledge is introduced. The authors try to claim that their paper focuses on the open-vocabulary setting, while UNITEXT does not. However, they do not make it clear what prevents the extension of UNITEXT to the open-vocabulary setting, and their particular design for such setting. In fact, as noted on UNITEXT's github project homepage, the approach extends easily to open-vocabulary setting and achieves better performance than the proposed approach on the Seginw benchmark. 2) The comparison with other open-vocabulary semantic/panoptic segmentation methods is unfair. Since the authors used many datasets with much overlap with ADE20k-150 or Pascal Context 59 for training, but not in the previous methods, I suspect that the performance gain achieved by the proposed method comes from the training dataset. The authors should evaluate their method on a harder dataset (e.g., ADE20K-full) and perform an analysis of the overlap between their training dataset and the test data.

Questions

See weakness.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

NaN

Summary

This paper presents HIPIE, an open vocabulary image segmentation model that produces segmentation from text prompts. The authors propose to decouple the segmentation of “thing” and “stuff” due to the differences in their semantic and geometric properties. By training on an additional part-level dataset, HIPIE can also perform part-level segmentation. The authors perform experiments on several open-vocabulary segmentation and referring image segmentation datasets and achieve better performances than previous state-of-the-art methods.

Strengths

- The decoupling of thing and stuff decoding makes sense because of the different feature distributions between the thing classes and the stuff classes. The authors have also experimentally verified that decoupling helps in quantitative measures. - The proposed method has strong performance on several popular datasets including COCO panoptic segmentation, ADE20K, the referring COCO dataset, etc. - The authors show that the proposed method can also work on part-level segmentation after being trained on part segmentation datasets. They show that the proposed method works better than Grounding DINO on at least one example in part segmentation which is an interesting application.

Weaknesses

- The paper claims a hierarchical representation which I find weak. There is only a small paragraph (Section 3.7) about hierarchical segmentation, and it is about part-level segmentation. From the text, it seems like the hierarchy is embedded in the text prompt (e.g., a head consists of ears, hair, etc.) – which any model that works on a text prompt is already capable of incorporating. The proposed model itself is not hierarchical. One thing that the proposed model does while prior work doesn’t is that it trains on part segmentation dataset (Pascal-Panoptic-Parts). However, it seems that the authors have only tested part segmentation performance on the same dataset, hence not demonstrating the “open vocabulary” capability. Overall, I find the claim of “hierarchical model” as a contribution confusing. - Most of the performance improvements seem to come from decoupling. It appears to me that decoupling leads to extra model parameters and run-time because there are two decoders to run for each input image. There are no comparisons with recent models in these two regards. - It is unclear why “feature fusion” should not be performed for the “stuff” branch.

Questions

Table 3 – what is the difference between the top part and the bottom part?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors present a section for limitations (Section A5) in the supplementary material. It points to several future work directions rather than limitations of the proposed model. One potential limitation is that the part-level segmentation might not generalize to vocabulary beyond the training set.

Summary

The paper proposes a unified method for open-vocabulary universal image segmentation and detection methods. A text-image fusion module takes both the image features and text features and then sends the fused results to the decoder. Several designed choices are presented and compared here. The model utilizes the dataset of Objects365, COCO, RefCOCO, RefCOCOg and RefCOCO+ for training and then test on different benchmarks.

Strengths

1. The overall method is clear and easy to understand. 2. The motivation to unify all the open-vocabulary segmentation and detection tasks is good.

Weaknesses

1. The model is first pretrained on Object365 which has over 600K well-labeled images and then finetuned on COCO, RefCOCO, RefCOCOg and RefCOCO+. Previous open-vocabulary image segmentation methods like [1-3] only trains on COCO dataset. I don't think the comparison is fair as the model has seen much more well-labeled datasets. 2. Since the hierarchical segmentation only requires the change of text prompts, I think all previous work can also perform on this task. Maybe the authors need to tone down the hierarchical thing and add some comparison with previous work. 3. The paper is titled open-vocabulary, however, lots of results in the paper are actually not such as the COCO results in Table 2, 5, 6. It's ok to report the results, but the authors need to report the open-vocabulary results which should be the main focus. 4. Some important comparisons are missing [4, 5] on open-vocabulary semantic segmentation. [1] Scaling Open-Vocabulary Image Segmentation with Image-Level Labels.\ [2] Open-vocabulary panoptic segmentation with maskclip.\ [3] Open-vocabulary panoptic segmentation with text-to-image diffusion models.\ [4] Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP.\ [5] Side Adapter Network for Open-Vocabulary Semantic Segmentation.

Questions

Table 3 seems confusing, why do the authors split the table into two parts?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes.

Summary

The paper propose to disentangle the representation learning and decoding for things and stuff and unify multiple segmentation tasks with different granularity (whole, part, subpart) and text formulation (reference text or category only). Extensive experiments are carried out to validate its effectiveness.

Strengths

1. The paper proposed to unifying different segmentation tasks and benchmark them, which is a promissing direction for future researches. 2. The decoupling of things and stuffs in representation learning are intuitive. 3. The experimental results are extensive and promissing.

Weaknesses

1. As the model only uses fully supervised data for supervision without large-scale image-text pretraining, whether its open-vocabulary capability is enough to be claimed as a open-vocabulary model? I am looking forward to see more open-vocabulary results that a quite different with exisitng training vocabulary with and without the assistance of CLIP for inference. 2. It seems that the architecture designs (decoupling thing and stuff) of this work do not feasible to the whole/part/subpart seg. If previous methods such as UNINEXT, X-Decoder, SEEM are also trained with part segmentation dataset (e.g., Pascal-Panoptic-Part). Can it already provide a solid results for such unified segmentation? Better with some experimental results.

Questions

Generally, I appreciate the setting of this work, but still have some concerns as detailed in the weakness part. If the author can address my concern, I am willing to upgrade my rating. Besides, I hope the author can have a detailed introduction about how to do the hierarchical segmentation in 3.7, you need to do the segmentation multiple times or just once? If the text query has both whole, part and subpart categories, how will the model do the inference.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

We sincerely appreciate your time and effort in reviewing our paper!

Dear Reviewer YNZH, We want to express our sincere gratitude for the time and effort you've dedicated to reviewing our paper. Your feedback has been invaluable in elevating the quality of our work. We're pleased to inform you that we've taken great care in addressing each of the questions and concerns you raised in your reviews. Here's a summary of our actions: - **Open-vocabulary results without CLIP**: We're excited to report that our HIPIE model, even without CLIP, ***outperforms the UNINEXT baseline and other con-current works***, such as OpenSeeD (ICCV2023), ***by a significant margin***. The results presented in Table R1 underscore our model's remarkable open vocabulary capability, demonstrating that it stands strong even without relying on CLIP assistance. - **Clarifications on hierarchical segmentation**: To provide better clarity, we've incorporated ***a diagram (Fig. R1 in the rebuttal PDF) that highlights the key differences*** from naively training the model on different granularities. Additionally, we've included ***qualitative results in Fig. R2*** and ***quantitative results in Table R2***, confirming the benefits of our design choice. Particularly, we've ***achieved an impressive increase of over 11.8 points in mIoUPartS*** compared to the UNINEXT baseline on the part-segmentation dataset. - **More open-vocabulary results**: In addition to the extensive evaluation results (***on over 40 datasets***) we provided in our main paper and appendix, we've included ***more results on A-150, A-847, CTX-459, and SeginW*** in the rebuttal. Notably, we've achieved state-of-the-art results on most of these benchmarks, showcasing our model's competitiveness (even comparing to con-current works)! We genuinely appreciate your insightful input, which has greatly influenced our revisions, and we're confident that the paper has significantly improved as a result. ***If you have any further questions or suggestions, we welcome your input!!!*** Your continued engagement is immensely valuable, and we look forward to ongoing discussions that will further enhance the revision of our paper. Warm regards, Paper 384 Authors

We sincerely thank you for your time and effort invested in reviewing our paper

Dear Reviewer EcWi, We sincerely thank you for your time and effort invested in reviewing our paper. Your feedback has proven to be invaluable in elevating the quality of our work. We're delighted to inform you that we've diligently addressed each of the questions and concerns you raised in your reviews. Here's a brief summary of our actions: - **HIPIE's pre-training on Object365**: To ensure a fair comparison, we've conducted a comprehensive evaluation (Table R1) against con-current works, including OpenSeed (ICCV2023), GroundingDINO (arXiv2023), and X-Decoder (CVPR2023), which employ datasets of notably larger scales than ours. We're thrilled to report that our HIPIE model ***outperforms not only the UNINEXT baseline but also these con-current works***, showcasing its significant capabilities. - **Clarifications on hierarchical segmentation**: To enhance clarity, we've included ***a diagram (Fig. R1 in the rebuttal PDF) illustrating the key distinctions*** from naively training the model on different granularities. Additionally, we've provided ***qualitative results in Fig. R2*** and ***quantitative results in Table R2***, demonstrating the advantages of our design choice. - **Comparing with UNINEXT trained on Part-Segmentation datasets**: We've significantly ***improved mIoUPartS by over 11.8 points*** compared to the UNINEXT baseline on the part-segmentation dataset. (Please see Table R2 for details) - **More open-vocabulary results**: In addition to the extensive evaluation results (***over 40 datasets***) presented in our main paper and appendix, we've included ***additional results on A-150, A-847, CTX-459, and SeginW*** in the rebuttal PDF. Notably, we've achieved state-of-the-art results on most of these benchmarks, highlighting our model's competitiveness, even when compared to con-current works. Furthermore, we're committed to addressing other minor questions related to table design. Your input has been instrumental in shaping our revisions, and we're confident that the paper has greatly benefited from your expertise. ***If you have any further questions or suggestions, please don't hesitate to share them!!!*** We will be happy to answer them! We eagerly anticipate further discussions that will enrich the revision of our paper. Warm regards, Paper 384 Authors

We truly appreciate your dedicated time and thoughtful review of our paper

Dear Reviewer 4C24, We truly appreciate your dedicated time and thoughtful review of our paper. Your feedback has been immensely valuable in enhancing the quality of our work. We're pleased to share that we have thoroughly addressed all the questions and concerns you raised in your reviews. Here's a concise summary of our actions: - **Clarifications on hierarchical segmentation**: We've included **a diagram (Fig. R1 in the rebuttal PDF) to illustrate the essential differences** from naively training the model on different granularities. Moreover, we've presented ***qualitative results in Fig. R2*** and ***quantitative results in Table R2***, effectively showcasing the benefits of our design choice. - **Comparison with works trained on Part-Segmentation datasets**: We've significantly ***boosted mIoUPartS by over 11.8 points*** in comparison to the UNINEXT baseline on the part-segmentation dataset (Please refer to Table R2 for detailed results). - **Impact of decoupling on model parameters and run-time**: We've addressed this concern by highlighting that ***the total parameters increased by only 4%*** compared to UNINEXT (805M vs 775M), leading to ***substantial performance gains*** and the ***introduction of new task capabilities***. Additionally, we've attended to other minor questions and aspects as needed. Your insights have greatly contributed to refining our work, and we believe the paper has significantly benefited from your expertise. ***If you have any further questions or suggestions, please don't hesitate to reach out!!!*** We're eager to provide any additional clarification needed! We look forward to continuing discussions that will enrich the revision of our paper. Warm regards, Paper 384 Authors

We genuinely appreciate the time and thought you invested in reviewing our paper

Dear Reviewer 7gze, We genuinely appreciate the time and thought you invested in reviewing our paper. Your feedback has been incredibly valuable in enhancing the quality of our work. We're pleased to inform you that we have carefully addressed all the questions and concerns you raised in your reviews. Here's a brief summary of our actions: - **Clarifications on hierarchical segmentation**: We've included a ***diagram (Fig. R1 in the rebuttal PDF) to illustrate the essential differences*** from naively training the model on different granularities. Additionally, we've presented ***qualitative results in Fig. R2*** and ***quantitative results in Table R2***, effectively showcasing the benefits of our design choice. - **Comparison with works trained on Part-Segmentation datasets**: We've ***significantly boosted mIoUPartS by over 11.8 points*** in comparison to the UNINEXT baseline on the part-segmentation dataset (Please refer to Table R2 for detailed results). - **Results comparison with UNINEXT on SeginW**: [***Please note that the results of UNINEXT on SeginW were released in June 2023, which is after NeurIPS deadline.***] Despite UNINEXT's strength in instance segmentation tasks, such as SeginW, ***UNINEXT's architecture lacks the capacity to execute panoptic and semantic segmentation*** effectively. In Table R1, we present the evaluation outcomes of UNINEXT when applied naively to ADE-150, ADE-Full, CTX-459, and SeginW benchmarks. ***Our approach outperforms UNINEXT by a large margin in instance segmentation, panoptic segmentation, and semantic segmentation.*** - **Overlap between training dataset and ADE-150 or Pascal Context**: We want to highlight that ***these datasets are commonly used in earlier works***, including OpenSeed. Additionally, we ***are not aware of any significant data overlap*** between the datasets we use for training and ADE-150, as well as Pascal Context. Your insights have significantly contributed to refining our work, and we believe the paper has greatly benefited from your expertise. ***If you have any further questions or suggestions, please don't hesitate to reach out!!*** We're eager to provide any additional clarifications needed! We look forward to continuing discussions that will enrich the revision of our paper. Warm regards, Paper 384 Authors