Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP

Summary

The paper tackles the open-vocabulary panoptic segmentation with a frozen CLIP. Unlike previous two-stages works, the paper uses the same backbone from the frozen CLIP for the mask generator and classifier. The single-stage framework accelerates the training and inferring speed and achieves SOTA performance.

Strengths

The single-stage framework proposed in the paper is concise and time-friendly for training and inference. The writing of the paper is clear and can be easily followed. The paper shows the convolutional CLIP has a better generalization ability to higher-resolution inputs than ViTs. FC-CLIP achieves SOTA performance on multiple open-vocabulary panoptic segmentation benchmarks.

Weaknesses

1. The novelty of the paper is weak. The idea of using the frozen CLIP for open-vocabulary prediction is the same as F-vlm. The main difference is that paper uses the framework of F-vlm to tackle the panoptic segmentation problem and substitute the Mask R-CNN heads in F-vlm with kMaX-DeepLab. Regarding innovation, I am not saying that the author must come up with some fancy innovations. But compared with the predecessors' work, what targeted improvements and discoveries have been made to new problems? Compared to F-vlm, I did not see any special design for the panoptic segmentation problem. 2. The claim that (frozen) CNN-based CLIP is better than ViT-based CLIP for dense prediction is not well verified. In the paper, the author verified the claim in Figure 1 with k-means visualization of these two types of CLIP. But whether the smoother feature correlates with the dense prediction performance is unclear. Many works [2][3] use ViT-based CLIP as the backbone of dense prediction tasks. Although the author compares ViT-based CLIP and CNN-based CLIP in Table 5 of Supp, the mask proposals are optimized with the CNN feature (on seen classes). It is more rigorous to use the ground truth mask to evaluate the classification performance of these two types of CLIP. The experiment of training a mask proposal on top of ViT-based CLIP is also needed to evaluate the mask proposal performance of ViT-based CLIP. The results of the 1281 resolution (used in the main paper) are missing in Table 5 of Supp. 3. Eq. 7 might have some typos. 'i' is the index of the mask, and 'i' should be substituted with the index of the class. Eq. 7 might be copied from F-vlm because in F-vlm, 'i' denotes the index of the class, which is correct. The author should check the formula more carefully. [1] Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-vlm: Open-vocabulary object detection upon frozen vision and language models. ICLR, 2023. [2] Ma, Chao et al. “Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models.” British Machine Vision Conference (2022). [3] Zhou, Ziqi et al. “ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation.” ArXiv abs/2212.03588 (2022): n. pag.

Questions

1. In Open-CLIP, they have three configures of ConvNext-Large: ConvNext-Large (D) @ 256x256 /w augreg on LAION-2B with a top-1 of 75.9% ConvNext-Large (D) @ 320x320 fine-tune of 256x256 weights above for ~2.5B more samples on LAION-2B, top-1 of 76.6% ConvNext-Large (D) @ 320x320 soup of 3 fine-tunes of 256x256 weights above on LAION-2B, top-1 of 76.9% Which one did the author use? The ConvNext-L trained with 320x320 is naturally more suitable for high-resolution inputs than ViT-L/14 trained with 224x224.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

In limitations, the author mentioned "how to deal with conflict or overlapping vocabularies (e.g. cat vs. cat head)". Does this problem occur in the experiment of the paper? What will happen to FC-CLIP if it faces the problem?

Summary

This paper proposes to build a one-stage open-vocabulary detector with a frozen language encoder (CLIP) and acvhieves good performance.

Strengths

1. Strong performance. Performance on many benchmarks is better than previous models. 2. Simple framework. One stage does look more simple to use.

Weaknesses

1. The novelty is limited. The proposed design does not involve enough novelty. 2. From my point of view, one stage and two stage methods do not differ that much. Why one-stage method is much better than previous methods?

Questions

Described above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Described above.

Summary

The authors propose an approach based on CLIP model and extend it for zero-shot semantic segmentation. The authors argue that pervious works solve the problem with a two-stage approach, which first generates mask predictions using one backbone and then extract features from another backbone using CLIP model is suboptimal and inefficient. They present an approach using a frozen CLIP image encoder as backbone and generate mask and prediction with trainable pixel and mask decoders. They validate the effectiveness of proposed method on several commonly used semantic segmentation benchmarks and achieve solid performance.

Strengths

1. Extending CLIP model to semantic segmentation in an efficient and effective approach is a recently popular and active research problem in Computer Vision. The authors present an approach to address this problem. 2. The proposed technique is technically sound. 3. The authors conduct experiments comparing to the latest previous work, ODISE, and shows that the performance is comparable while enjoy faster inference speed.

Weaknesses

1. The contribution is incremental. The proposed approach in very similar to OpenSeg [28], with just different selection of backbone, mask generation and classification of in-vocabulary classification. The minor difference in the selections of components is not significant for the top-tier conference. Comparing to [28], it is unclear where the improvement come from. It could be because of the selection of pixel decoder features for in-vocabulary categories, or the different selection of mask generation or the input resolution. There is no ablation to understand this and neither of these changes are significant contributions. 2. The naive single-stage baseline presented in the paper is not reasonable. As CLIP backbone is fine-tuned without considering text encoder, it naturally loses the capability of open-vocabulary classification. The baseline is not reasonable but created in a way to makes the contributions of proposed method appear meaningful 3. The technical details presented in this paper are not self-contained. Details as follow: - The description of naive single-stage baseline is vague. What is the mask generator used in the baseline? Does the classifier use text embedding or something else? - The description of class-agnostic mask generator presented in the paragraph starting at line 192 is vague and lack of details. What is the pixel decoder with axial attention? What is the kMax mask decoders? What is the k-means cross-attention? How is Hungarian matching used? How is the subset of predicted masks selected during the matching process? As mask generation is a critical component in the proposed method, it needs to be properly described even if they are introduced in the previous work to make the paper self-contained and to justify the contribution of the proposed method. - Abuse of notation. In line 136, sum{m_i} <= 1^{HxW} is not properly defined. I am guessing it means every entry in the matrix is smaller or equal to 1. This is a minor issue.

Questions

1. Please clarify the contributions of proposed methods. 2. Please justify the different between proposed approach and the previous work [28]

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The limitation is discussed in the paper to some extent and I don't have major concerns on the limitation.

Summary

This work proposes a new approach to open-vocabulary panoptic segmentation that unifies the mask generator and CLIP classifier into a single-stage framework. This is achieved by sharing the feature extractor between them, which presents two challenges: disrupting the alignment between image and text features during fine-tuning and the need for higher resolution inputs for dense prediction tasks. The authors address these challenges by adopting the shared Frozen CNN-based CLIP backbone. The resulting FC-CLIP model achieves state-of-the-art performance on several benchmarks while being more efficient and effective than previous methods.

Strengths

1. The paper's approach to unifying the mask generator and CLIP classifier into a single-stage framework is a novel contribution to the field of open-vocabulary panoptic segmentation. 2. The paper is well-written and clearly presents the problem of open-vocabulary panoptic segmentation, the challenges of the current two-stage pipeline, and the proposed FC-CLIP model. 3. The paper is well-organized and clearly presents the problem, methodology, and results. The authors provide a detailed explanation of the FC-CLIP model, making it easy to understand the proposed approach. The paper's figures and tables are well-designed and provide a clear visualization of the results.

Weaknesses

1. Absence of Ablation Study: The paper lacks an ablation study to conduct a thorough analysis of the impact of different components or design choices of the FC-CLIP model on its performance. For instance, the inclusion of an in/out-vocabulary classifier and the combine strategy, if adopted within the naive single-stage framework, what would be its effect on performance? 2. The experimental results are not convincing enough. The FC-CLIP model has a higher number of trainable parameters compared to ODISE. However, it remains uncertain whether the observed performance improvement can be attributed solely to the increased parameter count. (2) Table 2 highlights significant improvements achieved by the FC-CLIP model on the Cityscapes dataset. It would be beneficial to provide additional explanations or discussions to elucidate the reasons behind these improvements. (3) While FC-CLIP demonstrates noticeable enhancements on the ADE20K dataset, the improvements on the COCO dataset appear to be comparatively smaller.

Questions

The proposed method is well-motivated and novel. However, some key experiments are lack. More details please refer to the weakness part.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have briefly discussed the limitations and future work of their work. However, there are some limitations required to discussed. The FC-CLIP model relies on a CLIP model pre-trained on the Internet data that may be biased, which calls for future research for calibration to avoid misuse. They could provide a more detailed discussion of these issues. For example, they could discuss the potential biases in the pre-trained CLIP model and how these biases could impact the performance of the FC-CLIP model. They could also discuss the potential negative societal impact of the FC-CLIP model, such as its use in surveillance systems or other applications that could infringe on privacy rights.

Summary

In this submission, the authors proposed a new method for open-vocabulary panoptic segmentation. In the open-vocabulary segmentation setting, the model is trained on seen category annotations and tested on unseen categories. The frozen features of CLIP/ALIGN have been demonstrated to be effective in new category generalization. To leverage the representation of CLIP, prior works typically use the two-stage framework, one stage forwards high-resolution images for mask generation with vanilla networks, and the other stage inputs low-resolution images for mask classification. The proposed a shared Frozen Convolutional CLIP backbone (FC-CLIP) which unifies the pipelines into a single stage. It exploits frozen CLIP with ConvNeXt image encoder as the backbone network. FC-CLIP is simple and yet effective, surpass prior state-of-the-art on many open-vocabulary segmentation tasks.

Strengths

1. This submission explains the motivation and method very well. I like Figure 2 very much, which clearly shows the differences between FC-CLIP and prior works. 2. The proposed FC-CLIP simplifies the multiple forwards of the image encoder into just one forward pass, which saves the computation cost. 3. The proposed model outperforms many prior works and more quantitative results are also provided in the supplementary material.

Weaknesses

1. In the related work, I would suggest authors discuss more about the relationship with F-VLM, which also uses the frozen Convolution CLIP as the shared image encoder. What are the major differences between the designs of F-VLM and FC-CLIP? 2. The ablation study on CLIP model type is missing. In Figure 1, the authors demonstrate that CNN-based CLIP should get better features than ViT-based CLIP. I truly appreciate that the authors did compare ViT-L/14 with ConvNeXt-L in Table 5 of the supplementary material. I would suggest authors also compares other CNN-based CLIP like R50x4, R50x16, R50x64 used in F-VLM. 3. Table 5 in supplementary material shows a very interesting experiment, which shows increasing resolution from 224->448, ViT-L still outperforms ConvNeXt-L. But when increasing resolution to 672, ViT-L/14 accuracy drops significantly, which is slightly counter intuitive. Is this PQ evaluated with geometric mean or not? And I would also recommend evaluating zero-shot ImageNet classification accuracy at different resolution inputs, which could better justify whether CNN-based CLIP outperforms ViT-based CLIP when changing resolutions. 4. Authors state "training model solely with ViT-based CLIP is infeasible", probability due to the GPU memory constraint. To compare more fairly with CNN-based CLIP, I would suggest authors use a sliding window to extract features for ViT-based CLIP, e.g. sliding a 224x224 or 336x336 window over 1024x1024 input image to extract the ViT-based CLIP features, which should still have the capability of zero-shot classification.

Questions

1. I noted authors used a different resolution setting, long side 1281. Is there any ablation on this design compared to Mask2Former 1024 short-side resize used in other works? Besides, regarding the inference time comparison, I would suggest authors use the same input size for all the models. 2. In Table 1 of supplementary material, I think the last row should be swapped with the second last row.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

N/A

Response to author

Thank the authors for the detailed rebuttal. I like it a lot. I am glad to see Mask2Former further improve the performance of FC-CLIP with less trainable parameters. They are both great frameworks but with some different implementation details. I also truly appreciate authors' effort in open-souce and reproducibility. I would like to raise my rating from Weak Accept to Accept. And please do include more discussions with F-VLM for the general audience.