Summary
This paper present MCLIP, which aims to finetune CLIP using image and mask data without semantic labels. The goal is to make its open-vocabulary recognition ability to be adapted to position-sensitive semantic segmentation tasks. MCLIP chooses to use SAM or feature clusters of DINO to obtain masks, which are class-agnostic but sometimes too small to have consistent semantics for finetuning. In addition, in order to achieve a trade-off between training stability and avoiding catastrophic forgetting, clustering uses the EMA version of finetuned image encoder.
Strengths
- The idea of this paper is interesting. Solving open-vocabulary semantic segmentation without semantic labels is an interesting topic.
- The experimental results show that the preformance of the proposed approach is good.
- Ablation studies and analysis are provided to help better understand of the role of each component and hyper-parameter.
Weaknesses
- What role does the text encoder play in the clustering process?
The key to the success of clustering lies in the existence of unlabeled data in a certain grouping relationship in the space. The mask representations f_M can be considered to meet such conditions. If k learnable object prompts are initialized randomly, then k text representations f_C are randomly scattered in the text space initially, unable to provide the information needed for clustering. If k learnable object prompts are initialized by predefined categories, then this should violate the experimental setup without semantic labels. To illustrate this problem, the authors need to provide an explanation and add a comparative experiment: What results can be obtained just by clustering with f_M?
- Why use clustering technology to solve the problem of too fine initial mask granularity?
Firstly, previous work (GroupViT, CVPR'22 and SegCLIP, ICML'23) has used a grouping block designed based on the idea of clustering in the process of fine-tuning CLIP. In contrast, the clustering used in MCLIP provides better masks initially than the image patches used in previous work and the clustering process of MCLIP is parameter-free. It may be necessary to conduct ablation experiments to prove which differences bring positive results. But overall, technically, the use of clustering may be not novel.
- In order to obtain class-agnostic masks, there is a class of experimental settings called open set semantic
segmentation (O3S, arXiv preprint arXiv:2307.02003, 2023) . Can the ”SAM initialization + clustering” scheme proposed by MCLIP obtain competitive performance in the O3S experimental setting?
- Table 1 shows that using MCLIP’s fine-tuned CLIP instead of the Frozen CLIP in the previous method improves the performance of the previous method’s zero-shot mask classification. Taking FC-CLIP as an example, MCLIP improves its zero-shot mask classification performance on ADE20K by 2.7. But compared to the final performance of FC-CLIP with two branches integrated, it is still 4.0 lower. So, in the case of keeping the complete inference strategy of the previous method, how much improvement can be achieved by using MCLIP’s fine-tuned CLIP instead of the Frozen CLIP?
- In the ablation experiment in Table 3a, does ’w/o Semantic Clustering’ mean fine-tuning with SAM-initialized masks directly without clustering? If so, why is the performance so poor? Without using clustering, it is equivalent to the number of cluster k is greater than the number of masks initialized by SAM on all images. However, the conclusion obtained from table 3b is that with the increase of k, the
performance tends to saturation, rather than decreasing to the level of ”w/o Semantic Clustering”.