Weaknesses
**Overview**
Using cross-modal similarity map from attention layers is explored in numerous CLIP-based segmentation settings [1-6] and simply applying same to the case of diffusion models is minimally novel. Weight-free mask proposals from Stable Diffusion is done in prior work [10] and other contributions (root token, positional bias) are highly unclear. The paper in current form has numerous flaws; two main discussed below.
**Missing method details**
* (Sec 3.2) What is the tokenization used in language encoder? BPE like CLIP? Doesn't this split individual words in sub-parts (in some cases)?
Using given example, what happens if "horse" is split into "ho", "rse" tokens? Please provide these details and explain the root token selection clearly taking into account nature of tokenization scheme.
* (Sec 3.3) "Specifically, if the text, after syntactic analysis, contains explicit direction clues, P will be a soft mask with values ranging from 1 to 0 along the given direction axis." - how is direction obtained from the text?
* Missing ablation on positional bias
**Missing Related Work**
* Missing related work on weakly supervised open-vocab segmentation. Please discuss these works (e.g. 1-6) that can perform similar task and identical settings. Also consider comparing to one of these works (e.g. row in Table 1, 2) using their pre-trained model.
* Cite and discuss prior work that uses Stable Diffusion for zero-shot referring segmentation [7, 9] (exact same task). Also compare to these baselines.
* Discuss prior work exploring discriminative abilities of Stable Diffusion.
1. Zhang, Yabo et al. “Associating Spatially-Consistent Grouping with Text-supervised Semantic Segmentation.”
2. Luo, Huaishao et al. “SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation.” (ICML 2022)
3. Ranasinghe, Kanchana et al. “Perceptual Grouping in Contrastive Vision-Language Models.” (ICCV 2023).
4. Xu, Jilan et al. “Learning Open-Vocabulary Semantic Segmentation Models From Natural Language Supervision.” (CVPR 2023)
5. Cha, Junbum et al. “Learning to Generate Text-Grounded Mask for Open-World Semantic Segmentation from Only Image-Text Pairs.” (CVPR 2023)
6. Mukhoti, Jishnu et al. “Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning.” (CVPR 2023)
7. Burgert, Ryan et al. “Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors.” (CVPR 2023).
8. Li, Alexander C. et al. “Your Diffusion Model is Secretly a Zero-Shot Classifier.” (ICCV 2023)
9. Karazija, Laurynas et al. “Diffusion Models for Zero-Shot Open-Vocabulary Segmentation.”
10. Tian, Junjiao et al. “Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion.”
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.