Summary
This paper proposed a novel data curation/argumentation process, named Concept Curation (CoCu), for language-supervised semantic segmentation. In the setting of language-supervised semantic segmentation (e.g. GroupViT), the network is trained with image-text contrastive loss on large-scale image-text pairs. Authors identified several issues in the vanilla data distribution of the original contrastive learning, e.g. semantic gap, and semantic bias. The proposed CoCu mitigates these issues and improves over prior work GroupViT under a controlled setting, showing a faster convergence rate and high accuracy.
Strengths
1. In the quantitative evaluation, authors re-implemented GroupViT and compare CoCu and GroupViT under the controlled experiment setting, i.e. 1024 batch size. The authors also report the accuracies on additional evaluation datasets, IN50, IN300, Cityscapes, etc. CoCu outperforms GroupViT on all the datasets by a margin. Although CoCu doesn't achieve state-of-the-art results on some datasets, it doesn't degrade the effectiveness of the method.
2. The proposed method is well-motivated. The web-crawled image-text pairs are indeed quite noisy. And GroupViT is also known to be bad at segmenting background classes like grass and sky. Moreover, from the visualization in Figure 3(b), the CoCu shows that could focus on background grasses instead of the foreground fox.
3. In the Table 2 ablation study, when only trained on CC3M, CoCu improves over GroupViT by a large margin on Pascal VOC. It is a very interesting result and justifies that CoCu speeds up the convergence.
Weaknesses
1. Insufficient qualitative comparison. Since one major claim of CoCu is that, compared with vanilla contrastive loss used in GroupViT, the proposed dataset curation mitigates semantic cap and semantic bias issues. So besides quantitative evaluation metric mIoU, more visualizations compared with GroupViT are expected. I would suggest authors add more visualizations in the supplementary materials if there is no space left in the main submission.
2. In the ablation table 3, the authors studied the effects of different components of CoCu. But the average mIoU may not be an insightful metric. Since different datasets have very different category vocabulary, authors may try to include a detailed ablation study table in the supplementary material and elaborate more on the results.
Questions
1. The boundary between the semantic gap and semantic bias is not that clear. At least from Figure 1, it's hard to tell the major differences between them. Maybe authors could define them more rigorously and provide more examples about them.
2. How long does it take to process the dataset? It would be informative for other readers who want to reproduce or follow up on the curation process.
3. In Table 2, CoCu also showed significant improvement on ImageNet50 and ImageNet300. But for these two datasets, the vocabularies are mainly foreground, it would be very interesting to show why CoCu could improve on these datasets as well.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.